In more detail, RdbSaver uses AlignedBuffer that writes into io::Sink in chunks of 4KB.
It's great for the direct file I/O, but bad for sockets that receive blocks of 4KB with garbage
at the end. I improved the code around this and actually simplified the logic, so now AlignedBuffer
is just another Sink that is passed into serializer when writing into files. When sending to
sockets a socket sink is passed instead.
Also many other unrelated changes grouped into this pretty big cr.
1. dashtable readability improvements.
2. Move methods from facade::ConnectionContext - into facade::Service,
make ConnectionContext a dumb object.
3. Optionally allow journal to be memory only (not backed up by a disk)
by using a ring buffer to store last k entries in each journal slice. Also renamed
journal_shard into journal_slice because journal has presence in each DF thread and not
only in its shards.
4. Introduce journal::Entry that will consolidate any store change that happens in the thread.
5. Introduce GetRandomHex utility function.
6. Introduce two hooks: ServerFamily::OnClose that is called when a connection is closed,
and ServerFamily::BreakOnShutdown that is called when process exits and any background fibers neet to
break early.
7. Pull some noisy info logs out of rdb_load class.
8. Snapshot class now has the ability to subscribe to journal changes, thus it can include concurrent changes into the snapshot.
Currently only journal::Op::VAL is supported (it's part of RDB format anyway).
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* docs: fix Daniel Lemire's name
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(server): readability cleanup
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* docs: actually fix the name attribution
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* feat(core): Added DenseSet & StringSet types with docs
- Improved documentation by adding labels to chain types & pointer tagging table
- Added potential improvements to the DenseSet types in the docs
- Added excalidraw save file for future editing
- Removed ambiguous overloading types
- Renamed iterators to be more clear
* feat(core): Cleaned up DenseSet and Docs
* feat(core): Made DenseSet more ergonomic
* feat(server): Integration of DenseSet into Server
- Integrated DenseSet with CompactObj and the Set Family commands
Signed-off-by: Braydn <braydn.moore@uwaterloo.ca>
When a connection migrates to another thread, it also being handled by a different io_uring.
That means we need to cancel any pending io requests we had in the old io uring and reestablish them
in the new one.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Before this change Dragonfly evicted items only when it was low on memory and had to grow its main dictionary.
It is not enough because in many cases Dragonfly can grow in memory even when the main dictionary does not grow.
For example, when its dictionary is less than 90% utilized, but the newly added objects require lots of memory.
In addition, the dashtable adds additional segments, when other segments have enough available slots to fill the rest of the free memory.
This change adds another layer of defense that allows evicting items even when dictionary segments are not full.
The eviction is still local with respect to a segment. On my tests it seems that it's much harder to cross maxmemory limit than before.
In addition, we tightened the heuristic that allowes the dashtable to grow. Now it takes into account the average bytes per item
in order to project how much memory the full table takes before adding to it new segments.
This really improves dashtable utilization.
There are still things to improve:
1. the eviction behavior is rough. Once an insert does the eviction it tries to free enough objects to bring memory back.
This may result in very spiky insertion/eviction patterns.
2. The eviction procedure, even though it's limited to a single segment, is quite heavy because it goes over all buckets
in the segment. This may result in weird artifacts where we evict just enough to be under the limit, then add and evict
again and so on.
3. Therefore, we may need a periodic eviction that will compliment this emergency eviction step.
Fixes#224 and partially addresses #256
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
