* fix: Fix `test_take_over_seeder`
There are a few issues with the test:
1. Not using the admin port, which could cause pause to deadlock
2. Not waiting for some of the `task`s (although that won't cause a
failure)
But also in the product code:
1. We used to `std::move()` the same pointer multiple times
2. We assigned to the same status object from multiple threads
Hopefully this fixes the test. It used to fail every ~100 attempts on my
machine, now it's been >1,000 and they all passed.
* add comments
* remove shard_ptr param
* chore: introduce a cool queue that gradually retires cool items
This PR introduces a new state in which the offloaded value is not freed from memory but instead stays
in the cool queue.
Upon Read we convert the cool value back to hot table and delete it from storage.
When we low on memory we retire oldest cool values until we are above the threshold.
The PR does not fully finish the feature but it is workable enough to start (load)testing.
Missing:
a) Handle Modify operations
b) Retire cool items in more cases where we are low on memory. Specifically, refrain from evictions as long as cool items exist.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* default serialization_max_chunk_size to 10 mb
* add test for big values
* small rename of enum to conform style guide
---------
Signed-off-by: kostas <kostas@dragonflydb.io>
* chore: simplify computation of used_mem_current
Before - each thread updated its own variable and then,
the global "used_mem_current" was updated by summing used memory from each thread.
Now, each thread updates used_mem_current directly. The code is simpler and also provides more precise
results more frequently.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Also, add the according API to compact object.
Now external objects can be in two states: Cool and Offloaded.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Our area of attack during concurrent transaction access is the call to DisarmInShard and DisarmInShardWhen, which only access is_armed - an atomic varible. It is not safe to arbitrarily call GetNamespace() if we write to it in InitBase
Solution: Don't write to it post first initialization
* chore: fix test_parser_memory_stats flakiness
1. Added a robust assert_eventually decorator for pytests
2. Improved the assertion condition in TieredStorageTest.BackgroundOffloading
3. Added total_uploaded stats for tiering that tells how many times offloaded values
were promoted back to RAM.
* chore: skip test_cluster_fuzzymigration
1. Moved CommandGenerator to thread scope - there is no need to maintain separate command generator per connection.
2. Added "done" metric - to know how much was done so far.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: clean up TaskQueue since we do not need multiple fibers for it
Implement TaskQueue as a wrapper around FiberQueue.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Before it was possible to issue several concurrent AsyncWrite requests.
But these are not atomic, which leads to replication stream corruption.
Now we wait for the previous request to finish before sending the next one.
ThrottleIfNeeded is now takes into account pending buffer size for throttling.
Fixes#3329
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Leave only connection memory usage in memory stats.
We should think how we can move it also to /metrics.
In addition, added a test verifying that redis parser memory
usage is tracked.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* fix replication test flag name for big values
* fix a bug that triggers ub when RegisterOnChange is called on flows that iterate over the callbacks and preempt
* add a stress test for big value serialization
Signed-off-by: kostas <kostas@dragonflydb.io>
* feature(hset_family): Add NX option to HSETEX
fixes dragonflydb#3265
Signed-off-by: Stepan Bagritsevich <bagr.stepan@gmail.com>
* refactor(hset_family): Fix returned value in the HSETEX command
Signed-off-by: Stepan Bagritsevich <bagr.stepan@gmail.com>
* refactor: Revert the changes of the returned value for the HSETEX command
Signed-off-by: Stepan Bagritsevich <bagr.stepan@gmail.com>
---------
Signed-off-by: Stepan Bagritsevich <bagr.stepan@gmail.com>
* serialize big slots in chunks
* allow preemption on large slots
* disable big entries serialization for RDB files
* add test
Signed-off-by: kostas <kostas@dragonflydb.io>
We divide the keyspace between connections in advance.
This allows easily cover chunks of a key space in a predictable manner without having overlapping traffic.
Excess traffic will just wrap around.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: improve dfly_bench stats
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* Update src/server/dfly_bench.cc
Co-authored-by: Shahar Mike <chakaz@users.noreply.github.com>
Signed-off-by: Roman Gershman <romange@gmail.com>
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Signed-off-by: Roman Gershman <romange@gmail.com>
Co-authored-by: Shahar Mike <chakaz@users.noreply.github.com>
* feat(namespaces): Initial support for multi-tenant #3050
This PR introduces a way to create multiple, separate and isolated
namespaces in Dragonfly. Each user can be associated with a single
namespace, and will not be able to interact with other namespaces.
This is still experimental, and lacks some important features, such as:
* Replication and RDB saving completely ignores non-default namespaces
* Defrag and statistics either use the default namespace or all
namespaces without separation
To associate a user with a namespace, use the `ACL` command with the
`TENANT:<namespace>` flag:
```
ACL SETUSER user TENANT:namespace1 ON >user_pass +@all ~*
```
For more examples and up to date info check
`tests/dragonfly/acl_family_test.py` - specifically the
`test_namespaces` function.
