Before this change Dragonfly evicted items only when it was low on memory and had to grow its main dictionary.
It is not enough because in many cases Dragonfly can grow in memory even when the main dictionary does not grow.
For example, when its dictionary is less than 90% utilized, but the newly added objects require lots of memory.
In addition, the dashtable adds additional segments, when other segments have enough available slots to fill the rest of the free memory.
This change adds another layer of defense that allows evicting items even when dictionary segments are not full.
The eviction is still local with respect to a segment. On my tests it seems that it's much harder to cross maxmemory limit than before.
In addition, we tightened the heuristic that allowes the dashtable to grow. Now it takes into account the average bytes per item
in order to project how much memory the full table takes before adding to it new segments.
This really improves dashtable utilization.
There are still things to improve:
1. the eviction behavior is rough. Once an insert does the eviction it tries to free enough objects to bring memory back.
This may result in very spiky insertion/eviction patterns.
2. The eviction procedure, even though it's limited to a single segment, is quite heavy because it goes over all buckets
in the segment. This may result in weird artifacts where we evict just enough to be under the limit, then add and evict
again and so on.
3. Therefore, we may need a periodic eviction that will compliment this emergency eviction step.
Fixes#224 and partially addresses #256
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Resolves#206. There some minor fixes left to do, but they demand improvements in helio:
1. Remove spurious socket configuration calls that fail on uds.
2. Remove incorrect port printing for uds listener (listener_interface.cc:79).
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
This statistic helps understanding how much Dragonfly memory grows via updating the existing value vs the new ones.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. No entries data is written into a journal yet.
2. Introduced a state machine to start and stop journal using a new auxillary command "dfly".
3. string_family currently calls an universal command Journal::RecordEntry that should
save the current key/value of that entry. Please note that we won't use it for all the operations
because for some it's more efficient to record the action itself than the final value.
4. no locking information is recorded yet so atomicity of multi-key operations is not preserved for now.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* test: add dragonfly_db fixture to it tests #199
Signed-off-by: Shmulik Klein <shmulik.klein@gmail.com>
* test: lint using flake8
Signed-off-by: Shmulik Klein <shmulik.klein@gmail.com>
* test: run dragonfly debug version as fallback
Signed-off-by: Shmulik Klein <shmulik.klein@gmail.com>
Docker release pipeline can not push to protected main branch using
the service token. This switches to PAT secret.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Related to #159. Before this change, rdb loading thread has been creating all the redis objects as well.
Now we separate rdb file parsing and objects creation. File parsing phase produces a load trace of one or more binary blobs.
Those blobs are then passed to the threads that are responsible to manage the objects.
The second phase is object creation based on the trace that was passed. Finally those binary blobs are destroyed.
As a result, each thread creates objects using the memory allocator it owns and memory stats become consistent.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Fix SendStringArr logic. Consolidate both string_view and string variants to using the same code.
2. Tighten the test framework so that it will test that the parser consumed the whole response.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Make it use vectorized send instead of concatenating everything into a single string.
2. vectorized SendStringArr could fail sending large arrays for lengths higher than 512 (returned EMSGSIZE).
We improved the implementation so it would send those arrays in chunks of 256 items.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
