* default serialization_max_chunk_size to 10 mb
* add test for big values
* small rename of enum to conform style guide
---------
Signed-off-by: kostas <kostas@dragonflydb.io>
* chore: simplify computation of used_mem_current
Before - each thread updated its own variable and then,
the global "used_mem_current" was updated by summing used memory from each thread.
Now, each thread updates used_mem_current directly. The code is simpler and also provides more precise
results more frequently.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Also, add the according API to compact object.
Now external objects can be in two states: Cool and Offloaded.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Our area of attack during concurrent transaction access is the call to DisarmInShard and DisarmInShardWhen, which only access is_armed - an atomic varible. It is not safe to arbitrarily call GetNamespace() if we write to it in InitBase
Solution: Don't write to it post first initialization
* chore: fix test_parser_memory_stats flakiness
1. Added a robust assert_eventually decorator for pytests
2. Improved the assertion condition in TieredStorageTest.BackgroundOffloading
3. Added total_uploaded stats for tiering that tells how many times offloaded values
were promoted back to RAM.
* chore: skip test_cluster_fuzzymigration
1. Moved CommandGenerator to thread scope - there is no need to maintain separate command generator per connection.
2. Added "done" metric - to know how much was done so far.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: clean up TaskQueue since we do not need multiple fibers for it
Implement TaskQueue as a wrapper around FiberQueue.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Before it was possible to issue several concurrent AsyncWrite requests.
But these are not atomic, which leads to replication stream corruption.
Now we wait for the previous request to finish before sending the next one.
ThrottleIfNeeded is now takes into account pending buffer size for throttling.
Fixes#3329
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Leave only connection memory usage in memory stats.
We should think how we can move it also to /metrics.
In addition, added a test verifying that redis parser memory
usage is tracked.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* fix replication test flag name for big values
* fix a bug that triggers ub when RegisterOnChange is called on flows that iterate over the callbacks and preempt
* add a stress test for big value serialization
Signed-off-by: kostas <kostas@dragonflydb.io>
* feature(hset_family): Add NX option to HSETEX
fixes dragonflydb#3265
Signed-off-by: Stepan Bagritsevich <bagr.stepan@gmail.com>
* refactor(hset_family): Fix returned value in the HSETEX command
Signed-off-by: Stepan Bagritsevich <bagr.stepan@gmail.com>
* refactor: Revert the changes of the returned value for the HSETEX command
Signed-off-by: Stepan Bagritsevich <bagr.stepan@gmail.com>
---------
Signed-off-by: Stepan Bagritsevich <bagr.stepan@gmail.com>
* serialize big slots in chunks
* allow preemption on large slots
* disable big entries serialization for RDB files
* add test
Signed-off-by: kostas <kostas@dragonflydb.io>
We divide the keyspace between connections in advance.
This allows easily cover chunks of a key space in a predictable manner without having overlapping traffic.
Excess traffic will just wrap around.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: improve dfly_bench stats
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* Update src/server/dfly_bench.cc
Co-authored-by: Shahar Mike <chakaz@users.noreply.github.com>
Signed-off-by: Roman Gershman <romange@gmail.com>
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Signed-off-by: Roman Gershman <romange@gmail.com>
Co-authored-by: Shahar Mike <chakaz@users.noreply.github.com>
* feat(namespaces): Initial support for multi-tenant #3050
This PR introduces a way to create multiple, separate and isolated
namespaces in Dragonfly. Each user can be associated with a single
namespace, and will not be able to interact with other namespaces.
This is still experimental, and lacks some important features, such as:
* Replication and RDB saving completely ignores non-default namespaces
* Defrag and statistics either use the default namespace or all
namespaces without separation
To associate a user with a namespace, use the `ACL` command with the
`TENANT:<namespace>` flag:
```
ACL SETUSER user TENANT:namespace1 ON >user_pass +@all ~*
```
For more examples and up to date info check
`tests/dragonfly/acl_family_test.py` - specifically the
`test_namespaces` function.
* remove helper.h and helper.cc and move them to AclFamily class
* remove global tables from acl_commands_def and move them to AclFamily
Signed-off-by: kostas <kostas@dragonflydb.io>
It's more efficient to offload raw blobs kept in CompactObject than decoded strings.
Unfortunately, it's also more complicated. The complexity arises around Read/Modify operations
that must convert a raw blob into a processed string. Moreover, Modify decodes the raw string, therefore
subsequent callbacks already see the decoded string.
Finally, this PR fixes the logic in NotifyFetched flow:
before it could skip uploading the modified value back to memory, which breaks correctness.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
