Dragonfly

A novel memory store that supports Redis and Memcached commands. For more detailed status of what's implemented - see below.

Features include:

1. High throughput reaching millions of QPS on a single node.
2. TLS support.
3. Pipelining mode.
4. A novel cache design, which does not require specifying eviction policies.
5. Memory efficiency that can save 20-40% for regular workloads and even more for cache like workloads

Building from source

I've tested the build on Ubuntu 21.04+. Requires: CMake, Ninja, boost, libunwind8-dev

sudo apt install ninja-build
sudo apt install libunwind-dev
sudo apt-get install libboost-all-dev

git clone --recursive https://github.com/romange/dragonfly
cd dragonfly && ./helio/blaze.sh -release
cd build-opt && ninja dragonfly

Running

./dragonfly --logtostderr

for more options, run ./dragonfly --help

Milestone - Source Available

API 1.0

• String family
  • SET
  • SETNX
  • GET
  • DECR
  • INCR
  • DECRBY
  • GETSET
  • INCRBY
  • MGET
  • MSET
  • MSETNX
  • SUBSTR
• Generic family
  • DEL
  • ECHO
  • EXISTS
  • EXPIRE
  • EXPIREAT
  • PING
  • RENAME
  • RENAMENX
  • SELECT
  • TTL
  • TYPE
  • SORT
• Server Family
  • QUIT
  • DBSIZE
  • BGSAVE
  • SAVE
  • DBSIZE
  • DEBUG
  • EXEC
  • FLUSHALL
  • FLUSHDB
  • INFO
  • MULTI
  • SHUTDOWN
  • LASTSAVE
  • SLAVEOF/REPLICAOF
  • SYNC
• Set Family
  • SADD
  • SCARD
  • SDIFF
  • SDIFFSTORE
  • SINTER
  • SINTERSTORE
  • SISMEMBER
  • SMOVE
  • SPOP
  • SRANDMEMBER
  • SREM
  • SMEMBERS
  • SUNION
  • SUNIONSTORE
• List Family
  • LINDEX
  • LLEN
  • LPOP
  • LPUSH
  • LRANGE
  • LREM
  • LSET
  • LTRIM
  • RPOP
  • RPOPLPUSH
  • RPUSH
• SortedSet Family
  • ZADD
  • ZCARD
  • ZINCRBY
  • ZRANGE
  • ZRANGEBYSCORE
  • ZREM
  • ZREMRANGEBYSCORE
  • ZREVRANGE
  • ZSCORE
• Not sure whether these are required for the initial release.
  • AUTH
  • BGREWRITEAOF
  • KEYS
  • MONITOR
  • RANDOMKEY
  • MOVE

API 2.0

• List Family
  • BLPOP
  • BRPOP
  • BRPOPLPUSH
  • BLMOVE
  • LINSERT
  • LPUSHX
  • RPUSHX
• String Family
  • SETEX
  • APPEND
  • PREPEND (dragonfly specific)
  • BITCOUNT
  • BITFIELD
  • BITOP
  • BITPOS
  • GETBIT
  • GETRANGE
  • INCRBYFLOAT
  • PSETEX
  • SETBIT
  • SETRANGE
  • STRLEN
• HashSet Family
  • HSET
  • HMSET
  • HDEL
  • HEXISTS
  • HGET
  • HMGET
  • HLEN
  • HINCRBY
  • HINCRBYFLOAT
  • HGETALL
  • HKEYS
  • HSETNX
  • HVALS
  • HSCAN
• PubSub family
  • PUBLISH
  • PUBSUB
  • PUBSUB CHANNELS
  • SUBSCRIBE
  • UNSUBSCRIBE
  • PSUBSCRIBE
  • PUNSUBSCRIBE
• Server Family
  • WATCH
  • UNWATCH
  • DISCARD
  • CLIENT KILL/LIST/UNPAUSE/PAUSE/GETNAME/SETNAME/REPLY/TRACKINGINFO
  • COMMAND
  • COMMAND COUNT/GETKEYS/INFO
  • CONFIG GET/REWRITE/SET/RESETSTAT
  • MIGRATE
  • ROLE
  • SLOWLOG
  • PSYNC
  • TIME
  • LATENCY...
• Generic Family
  • SCAN
  • PEXPIREAT
  • PEXPIRE
  • DUMP
  • EVAL
  • EVALSHA
  • OBJECT
  • PERSIST
  • PTTL
  • RESTORE
  • SCRIPT LOAD
  • SCRIPT DEBUG/KILL/FLUSH/EXISTS
• Set Family
  • SSCAN
• Sorted Set Family
  • ZCOUNT
  • ZINTERSTORE
  • ZLEXCOUNT
  • ZRANGEBYLEX
  • ZRANK
  • ZREMRANGEBYLEX
  • ZREMRANGEBYRANK
  • ZREVRANGEBYSCORE
  • ZREVRANK
  • ZUNIONSTORE
  • ZSCAN
• HYPERLOGLOG Family
  • PFADD
  • PFCOUNT
  • PFMERGE

In addition, we want to support efficient expiry (TTL) and cache eviction algorithms. We should implement basic memory management support. For Master/Slave replication we should design a distributed log format.

Memchache API

• set
• get
• replace
• add
• stats (partial)
• append
• prepend
• delete
• flush_all
• incr
• decr
• version
• quit

Commands that I prefer avoid implementing before launch:

• PUNSUBSCRIBE
• PSUBSCRIBE
• HYPERLOGLOG
• SCRIPT DEBUG
• OBJECT
• DUMP/RESTORE
• CLIENT

Also, I would omit keyspace notifications. For that I would like to deep dive and learn exact use-cases for this API.

Random commands we implemented as decorators along the way

• ROLE (2.8) decorator for for master withour replicas
• UNLINK (4.0) decorator for DEL command
• BGSAVE
• FUNCTION FLUSH

Milestone Stability

APIs 3,4,5 without cluster support, without modules, without memory introspection commands. Without geo commands and without support for keyspace notifications, without streams. Design config support. ~10-20 commands overall... Probably implement cluster-API decorators to allow cluster-configured clients to connect to a single instance.

• HSTRLEN

Design decisions along the way

Expiration deadlines with relative accuracy

I decided to limit the expiration range to 365 days. Moreover, expiration deadlines with millisecond precision (PEXPIRE/PSETEX etc) will be rounded to closest second **for deadlines greater than 33554431ms (approximately 560 minutes). In other words, expiries of PEXPIRE key 10010 will expire exactly after 10 seconds and 10ms. However, PEXPIRE key 34000300 will expire after 34000 seconds (i.e. 300ms earlier). Similarly, PEXPIRE key 34000800 will expire after 34001 seconds, i.e. 200ms later.

Such rounding has at most 0.002% error which I hope is acceptable for large ranges. If it breaks your use-cases - talk to me or open an issue and explain your case.

For more detailed differences between this and Redis implementations see here.