Sometimes for large values during snapshot loading/saving we allocate a lot of extra memory. For that, we might need to manually run memory decommit for mimalloc to release memory pages back to the OS. This PR addresses that by manually running memory decommit after each shard finishes loading or saving a snapshot.
---------
Signed-off-by: kostas <kostas@dragonflydb.io>
* chore: ClearInternal now can clear partially
Intended for future use - to deallocate large objects gradually.
Currently nothing is changed in the functionality besides some cleanups.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: Implement AddMany method
1. Fix a performance bug in Find2 that made redundant comparisons
2. Provide a method to StringSet that adds several items in a batch
3. Use AddMany inside set_family
Before:
```
BM_Add 4253939 ns 4253713 ns 991
```
After:
```
BM_Add 3482177 ns 3482050 ns 1206
BM_AddMany 3101622 ns 3101507 ns 1360
```
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: fixes
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Clean up interface a bit. AddOrFindDense does not make much sense as a single function
because it does not provide any performance benefits - we still must perform a lookup
before inserting. AddSds should have been removed a long time ago.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* feat: add DenseSet::IteratorBase::SetExpiryTime
This commit is in preparation for adding FIELDEXPIRE and HEXPIRE.
* fix: 0 is a valid input for MakeSetSds
* feat(rdb_load): add support for loading huge hmaps
* feat(rdb_load): add support for loading huge zsets
* feat(rdb_load): log DFATAL when append fails
A common case is that we need to clean up a connection before we exit a test via .close() method. This is needed because otherwise the connection will raise a warning that it is left unclosed. However, remembering to call .close() at each connection at the end of the test is cumbersome! Luckily, fixtures in python can be marked as async which allow us to:
* cache all clients created by DflyInstance.client()
* clean them all at the end of the fixture in one go
---------
Signed-off-by: kostas <kostas@dragonflydb.io>
We do not allow notify_keyspace_events to be set at runtime via config set command.
* allow notify_keyspace_events in config set command
* add tests
---------
Signed-off-by: kostas <kostas@dragonflydb.io>
Seems that lpStringToInt64 is fairly optimized.
On c7g:
```
BM_LpString2Int/1 27383 ns 27381 ns 204149
BM_LpString2Int/2 24535 ns 24534 ns 227981
```
so SimpleAtoi has only ~10% improvement.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: introduce an optimized integer compare algorithm for lists
Problem: when lists are long OpRem will spend lots of time comparing an element with records in the list.
For integer-based lists, most of the time is spent in lpCompare.
In addition, lpGet has lots of branches that penalize integers use-cases.
This PR:
1. Introduces lpGet2 - the improved version with less branches.
2. Benchmarks lpCompare vs an algorithm that compares records to an integer.
3. Benchmarks lpGet vs lpGet2
```
----------------------------------------------------------
Benchmark Time CPU Iterations
----------------------------------------------------------
BM_LpCompare 1187 ns 1187 ns 4715144
BM_LpCompareInt 371 ns 371 ns 15216611
BM_LpGet/1 265 ns 265 ns 21473149
BM_LpGet/2 214 ns 214 ns 26075164
```
There are no functional changes to the Dragonfly code.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: fixes
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Previously XREAD:
- Fetched group consumer info (with a hop)
- Looked up last stream ids (with a hop)
- Determined if there are entries to read (on coordinator)
- Dispatched a hop to read entries or retired to blocking mode
Instead we can merge steps (1), (2) and (3) into a single step, optionally with step (4) for single shard operations that can avoid concluding (we had this optimization before)
1. Issue ping upon connect, add a comment why.
2. log error if dfly_bench disconnects before all the requests were processed.
3. Refactor memcache parsing code into ParseMC function.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
A small refactoring to improve the flow of ScheduleInternal() as well as
to prepare it for the next change that will reduce the CPU load from the shard queue.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* fix bitiop creating the dst key if result is empty
* fix replicating dst with the wrong type
* make bitop a blind update (similar to set command)
---------
Signed-off-by: kostas <kostas@dragonflydb.io>
* chore: Forbid replicating a replica
We do not support connecting a replica to a replica, but before this PR
we allowed doing so. This PR disables that behavior.
Fixes#3679
* `replicaof_mu_`
fix: Fix flaky test `test_acl_revoke_pub_sub_while_subscribed`
The reason it failed is that, in some rare cases, the subscriber did not
get the first few messages of the publisher. This is likely due to
timing of subscribe and publish, in different connections / threads.
Given Pub/Sub has very weak guarantees, it's probably ok as is, so I
just added a sleep to get the test to pass always.
