Dragonfly

A novel memory store that supports Redis and Memcached commands. For more detailed status of what's implemented - see below.

Features include:

High throughput reaching millions of QPS on a single node.
TLS support.
Pipelining mode.
A novel cache design, which does not require specifying eviction policies.
Memory efficiency that can save 20-40% for regular workloads and even more for cache like workloads

Building from source

I've tested the build on Ubuntu 21.04+. Requires: CMake, Ninja, boost, libunwind8-dev

sudo apt install ninja-build
sudo apt install libunwind-dev
sudo apt-get install libboost-all-dev

git clone --recursive https://github.com/romange/dragonfly
cd dragonfly && ./helio/blaze.sh -release
cd build-opt && ninja dragonfly

Running

./dragonfly --logtostderr

for more options, run ./dragonfly --help

Milestone Egg 🥚

API 1.0

String family
- SET
- SETNX
- GET
- DECR
- INCR
- DECRBY
- GETSET
- INCRBY
- MGET
- MSET
- MSETNX
- SUBSTR
Generic family
- DEL
- ECHO
- EXISTS
- EXPIRE
- EXPIREAT
- PING
- RENAME
- RENAMENX
- SELECT
- TTL
- TYPE
- SORT
Server Family
- QUIT
- DBSIZE
- BGSAVE
- SAVE
- DBSIZE
- DEBUG
- EXEC
- FLUSHALL
- FLUSHDB
- INFO
- MULTI
- SHUTDOWN
- LASTSAVE
- SLAVEOF/REPLICAOF
- SYNC
Set Family
- SADD
- SCARD
- SDIFF
- SDIFFSTORE
- SINTER
- SINTERSTORE
- SISMEMBER
- SMOVE
- SPOP
- SRANDMEMBER
- SREM
- SMEMBERS
- SUNION
- SUNIONSTORE
List Family
- LINDEX
- LLEN
- LPOP
- LPUSH
- LRANGE
- LREM
- LSET
- LTRIM
- RPOP
- RPOPLPUSH
- RPUSH
SortedSet Family
- ZADD
- ZCARD
- ZINCRBY
- ZRANGE
- ZRANGEBYSCORE
- ZREM
- ZREMRANGEBYSCORE
- ZREVRANGE
- ZSCORE
Not sure whether these are required for the initial release.
- AUTH
- BGREWRITEAOF
- KEYS
- MONITOR
- RANDOMKEY
- MOVE

In addition, we want to support efficient expiry (TTL) and cache eviction algorithms. We should implement basic memory management support. For Master/Slave replication we should design a distributed log format.

Memchache API

set
get
replace
add
stats (partial)
append
prepend
delete
flush_all
incr
decr
version
quit

API 2.0

List Family
- BLPOP
- BRPOP
- BRPOPLPUSH
- BLMOVE
- LINSERT
- LPUSHX
- RPUSHX
String Family
- SETEX
- APPEND
- PREPEND (dragonfly specific)
- BITCOUNT
- BITFIELD
- BITOP
- BITPOS
- GETBIT
- GETRANGE
- INCRBYFLOAT
- PSETEX
- SETBIT
- SETRANGE
- STRLEN
HashSet Family
- HSET
- HMSET
- HDEL
- HEXISTS
- HGET
- HMGET
- HLEN
- HINCRBY
- HINCRBYFLOAT
- HGETALL
- HKEYS
- HSETNX
- HVALS
- HSCAN
PubSub family
- PUBLISH
- PUBSUB
- PUBSUB CHANNELS
- SUBSCRIBE
- UNSUBSCRIBE
Server Family
- WATCH
- UNWATCH
- DISCARD
- CLIENT KILL/LIST/UNPAUSE/PAUSE/GETNAME/SETNAME/REPLY/TRACKINGINFO
- COMMAND
- COMMAND COUNT/GETKEYS/INFO
- CONFIG GET/REWRITE/SET/RESETSTAT
- MIGRATE
- ROLE
- SLOWLOG
- PSYNC
- TIME
- LATENCY...
Generic Family
- SCAN
- PEXPIREAT
- PEXPIRE
- DUMP
- EVAL
- EVALSHA
- OBJECT
- PERSIST
- PTTL
- RESTORE
- SCRIPT LOAD
- SCRIPT DEBUG/KILL/FLUSH/EXISTS
Set Family
- SSCAN
- SMISMEMBER
Sorted Set Family
- ZCOUNT
- ZINTERSTORE
- ZLEXCOUNT
- ZRANGEBYLEX
- ZRANK
- ZREMRANGEBYLEX
- ZREMRANGEBYRANK
- ZREVRANGEBYSCORE
- ZREVRANK
- ZUNIONSTORE
- ZSCAN
HYPERLOGLOG Family
- PFADD
- PFCOUNT
- PFMERGE
PUBSUB Family
- PSUBSCRIBE
- PUBSUB
- PUBLISH
- PUNSUBSCRIBE
- SUBSCRIBE
- UNSUBSCRIBE

Commands that I prefer avoid implementing before launch:

PUNSUBSCRIBE
PSUBSCRIBE
HYPERLOGLOG
SCRIPT DEBUG
OBJECT
DUMP/RESTORE
CLIENT

Also, I would omit keyspace notifications. For that I would like to deep dive and learn exact use-cases for this API.

Random commands we implemented along the way

ROLE (2.8) decorator for for master withour replicas
UNLINK (4.0) decorator for DEL command
BGSAVE

Milestone Nymph

API 2,3,4 without cluster support, without modules, without memory inspection commands. Without support for keyspace notifications.

Design config support. ~140 commands overall...

Milestone Molt

API 5,6 - without cluster and modules. Streams support. ~80 commands overall.

Milestone Adult

TBD.

Design decisions along the way

Expiration deadlines with relative accuracy

I decided to limit the expiration range to 365 days. Moreover, expiration deadlines with millisecond precision (PEXPIRE/PSETEX etc) will be rounded to closest second **for deadlines greater than 33554431ms (approximately 560 minutes). In other words, expiries of PEXPIRE key 10010 will expire exactly after 10 seconds and 10ms. However, PEXPIRE key 34000300 will expire after 34000 seconds (i.e. 300ms earlier). Similarly, PEXPIRE key 34000800 will expire after 34001 seconds, i.e. 200ms later.

Such rounding has at most 0.002% error which I hope is acceptable for large ranges. If it breaks your use-cases - talk to me or open an issue and explain your case.

6 KiB Raw Blame History