1
0
Fork 0
mirror of https://github.com/dragonflydb/dragonfly.git synced 2024-12-15 17:51:06 +00:00
Commit graph

643 commits

Author SHA1 Message Date
Roman Gershman
cfd2273fb0
chore: improve replication locks (#3436)
* chore: improve replication locks

Allow non-exclusive, read-only access to Dfly::ReplicaInfo structure.
The most important change is in DflyCmd::CancelReplication, where before
it has locked ReplicaInfo mutex and then continued with locking the global mutex.
It is dangerous because most operation lock them in the opposite order.

Also rename ambigous GetReplicaInfo accessors to clearer names.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>

* chore: comments

* chore: comments

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-08-04 10:55:50 +00:00
Vladislav
2ef475865f
test(cluster): Migration replication test (#3417) 2024-08-04 12:45:02 +03:00
Shahar Mike
2aa0b70035
feat(server): Support replica-announce-ip/port (#3421)
* feat: Support `replica-announce-ip`/`port`

Before this PR, we only supported `cluster_announce_ip`.
It's basically the same feature, but used for cluster announcements
instead of replication.

This PR adds support for `replica-announce-ip` and
`replica-announce-port`, which can be set via new flags `--announce_ip=`
and `--announce_port=`. These flags apply to both cluster and replica
announcements.

Tested via running Sentinel, and making sure it is able to connect to
announced ip+port, while it can't connect to announced false /
unavailable ip+port.

Note: this PR deprecates `--cluster_announce_ip`, but continues to
support it. We will remove it in a future version.

Fixes #3380

* fix failing test

* destructure
2024-08-04 12:35:14 +03:00
Vladislav
82298b8122
fix(server): Implement SCRIPT GC command (#3431)
* fix(server): Implement SCRIPT GC command
2024-08-02 23:49:51 +03:00
Borys
e2b6cfb384
chore: skip cluster tests if redis-server wasn't found (#3416)
* chore: skip cluster tests if redis-server wasn't found
2024-08-01 13:04:02 +00:00
Roman Gershman
0ad310717d
chore: Tiered fixes (#3401)
1. Add background offloading stats
2. remove direct_fd override - helio is already updated with default=false, so it's not needed anymore.
3. remove redundant tiered_storage_memory_margin flag

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-08-01 11:03:13 +03:00
Roman Gershman
71b861572a
chore: remove verbose printing of tests (#3420)
Motivation: to avoid 80MB logs into stdout like this one:
https://github.com/dragonflydb/dragonfly/actions/runs/10174852001/job/28141278813?pr=3401

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-08-01 10:52:50 +03:00
Vladislav
e273015c0b
fix(connection): Count memchached pipelined commands (#3413) 2024-08-01 10:14:36 +03:00
Kostas Kyrimis
aa02070e3d
chore: add db_slice lock to protect segments from preemptions (#3406)
DastTable::Traverse is error prone when the callback passed preempts because the segment might change. This is problematic and we need atomicity while traversing segments with preemption. The fix is to add Traverse in DbSlice and protect the traversal via ThreadLocalMutex.

* add ConditionFlag to DbSlice
* add Traverse in DbSlice and protect it with the ConditionFlag
* remove condition flag from snapshot
* remove condition flag from streamer

---------

Signed-off-by: kostas <kostas@dragonflydb.io>
2024-07-30 15:02:54 +03:00
Vladislav
f536f8afbd
chore: cancel slot migrations on shutdown (#3405) 2024-07-30 12:47:58 +03:00
Shahar Mike
20bda84317
Revert "chore: set serialization_max_chunk_size to 1 byte (#3379)" (#3398)
This reverts commit 2867d54a05.
2024-07-28 06:48:46 +00:00
Stepan Bagritsevich
28cfde0a27
fix: Fix unsupported object type rejson-rl in RedisInsight (#3384)
* fix: Fix unsupported object type rejson-rl in RedisInsight

Signed-off-by: Stepan Bagritsevich <bagr.stepan@gmail.com>

* fix(generic_family): fix case for the TYPE option in SCAN command

Signed-off-by: Stepan Bagritsevich <bagr.stepan@gmail.com>

* feat(generic_family_test): Add test for the Redis GUI

Signed-off-by: Stepan Bagritsevich <bagr.stepan@gmail.com>

* refactor: address comments

Signed-off-by: Stepan Bagritsevich <bagr.stepan@gmail.com>

* refactor: address comments 2

Signed-off-by: Stepan Bagritsevich <bagr.stepan@gmail.com>

* refactor: change variable name from obj_type_as_string to obj_type

Signed-off-by: Stepan Bagritsevich <bagr.stepan@gmail.com>

---------

Signed-off-by: Stepan Bagritsevich <bagr.stepan@gmail.com>
2024-07-27 19:05:00 +02:00
Kostas Kyrimis
2867d54a05
chore: set serialization_max_chunk_size to 1 byte (#3379)
Update the flag for extreme testing. We should remove this before the release.

* set serialization_max_chunk_size to 1 byte

---------

Signed-off-by: kostas <kostas@dragonflydb.io>
2024-07-25 23:10:44 +03:00
Kostas Kyrimis
6d9e370e2d
fix: test_big_value_serialization_memory_limit shutdown timeout (#3390)
The problem is that the test test_big_value_serialization_memory_limit will try to shutdown dragonfly at the end with a timeout of 15 seconds. Dragonfly during shutdown takes a snapshot which might take more than 15 seconds and the test fails.

* call flushall before we exit the test

---------

Signed-off-by: kostas <kostas@dragonflydb.io>
2024-07-25 21:38:09 +03:00
Shahar Mike
fb4222d01e
fix: Fix test_take_over_seeder (#3385)
* fix: Fix `test_take_over_seeder`

There are a few issues with the test:

1. Not using the admin port, which could cause pause to deadlock
2. Not waiting for some of the `task`s (although that won't cause a
   failure)

But also in the product code:

1. We used to `std::move()` the same pointer multiple times
2. We assigned to the same status object from multiple threads

Hopefully this fixes the test. It used to fail every ~100 attempts on my
machine, now it's been >1,000 and they all passed.

* add comments

* remove shard_ptr param
2024-07-25 08:00:05 +00:00
Kostas Kyrimis
929222a7df
chore: add mem test for big values and default the flag (#3369)
* default serialization_max_chunk_size to 10 mb
* add test for big values
* small rename of enum to conform style guide

---------

Signed-off-by: kostas <kostas@dragonflydb.io>
2024-07-24 16:07:27 +03:00
Kostas Kyrimis
cd863b89b4
chore: disable cluster_fuzzymigration (#3373)
* mark cluster_fuzzymigration as skipped

Signed-off-by: kostas <kostas@dragonflydb.io>
2024-07-24 11:46:44 +03:00
Kostas Kyrimis
cd0e03a737
chore: disable compression on big values (#3358)
*  compression when we chunk big values

Signed-off-by: kostas <kostas@dragonflydb.io>
2024-07-23 08:57:21 +00:00
Roman Gershman
aac90f25b5
fix: failure in test_cluster_fuzzymigration (#3363) 2024-07-22 22:39:41 +03:00
Roman Gershman
4b1574b5c8
chore: fix test_parser_memory_stats flakiness (#3354)
* chore: fix test_parser_memory_stats flakiness

1. Added a robust assert_eventually decorator for pytests
2. Improved the assertion condition in TieredStorageTest.BackgroundOffloading
3. Added total_uploaded stats for tiering that tells how many times offloaded values
   were promoted back to RAM.

* chore: skip test_cluster_fuzzymigration
2024-07-22 10:41:26 +00:00
Roman Gershman
feb9bc266a
chore: pull helio (#3350)
* chore: pull helio

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-07-21 15:26:25 +03:00
Roman Gershman
fb7782bcce
chore: remove redundant metrics from memory stats (#3345)
Leave only connection memory usage in memory stats.
We should think how we can move it also to /metrics.
In addition, added a test verifying that redis parser memory
usage is tracked.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-07-20 06:02:55 -04:00
Shahar Mike
2b54fd985f
fix: Cancel outgoing migration when retrying / closing (#3339) 2024-07-19 07:49:49 +00:00
Kostas Kyrimis
8a2d6ad1f4
fix: ub in RegisterOnChange and regression tests for big values (#3336)
* fix replication test flag name for big values
* fix a bug that triggers ub when RegisterOnChange is called on flows that iterate over the callbacks and preempt
* add a stress test for big value serialization

Signed-off-by: kostas <kostas@dragonflydb.io>
2024-07-19 07:03:17 +00:00
Borys
cad62679a4
Fix blocking commands moved error (#3334)
* fix: BLPOP BZPOP(MIN|MAX) moved error
2024-07-18 20:38:13 +03:00
Kostas Kyrimis
bfa5df5d6c
feat: add an option to flush serialized entries on threshold limit (#3241)
* serialize big slots in chunks
* allow preemption on large slots
* disable big entries serialization for RDB files
* add test

Signed-off-by: kostas <kostas@dragonflydb.io>
2024-07-18 10:15:41 +00:00
Borys
1acc824eff
fix(test): copy logs for failed test during TEARDOWN phase (#3331)
* fix(test): copy logs for failed test during TEARDOWN phase
2024-07-17 22:16:08 +03:00
Shahar Mike
18ca61d29b
feat(namespaces): Initial support for multi-tenant (#3260)
* feat(namespaces): Initial support for multi-tenant #3050

This PR introduces a way to create multiple, separate and isolated
namespaces in Dragonfly. Each user can be associated with a single
namespace, and will not be able to interact with other namespaces.

This is still experimental, and lacks some important features, such as:
* Replication and RDB saving completely ignores non-default namespaces
* Defrag and statistics either use the default namespace or all
  namespaces without separation

To associate a user with a namespace, use the `ACL` command with the
`TENANT:<namespace>` flag:

```
ACL SETUSER user TENANT:namespace1 ON >user_pass +@all ~*
```

For more examples and up to date info check
`tests/dragonfly/acl_family_test.py` - specifically the
`test_namespaces` function.
2024-07-16 19:34:49 +03:00
Borys
3891efac2c
fix: forbid DFLYCLUSTER commads set for emulated cluster mode (#3307)
* fix: forbid DFLYCLUSTER commads set for emulated cluster mode
* feat: add CLUSTER MYID and remove DFLYCLUSTER MYID
* fix(test): __del__ method in python can't be async
* fix: crash and test_replicate_disconnect_cluster
2024-07-16 14:17:28 +03:00
adiholden
f20318d88a
fix (pytest): generate unique random dbfilename for tests (#3317)
Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-07-15 11:01:02 +03:00
Kostas Kyrimis
bf1b6cef6e
chore: skip test_cluster_flushall_during_migration (#3316)
* skip failing test on ci

Signed-off-by: kostas <kostas@dragonflydb.io>
2024-07-15 09:48:28 +03:00
Kostas Kyrimis
73abd68f2d
chore(acl): add test with requirepass and aclfile (#3312)
* add a test that uses both requirepass and aclfile

Signed-off-by: kostas <kostas@dragonflydb.io>
2024-07-15 08:28:29 +03:00
Shahar Mike
1daa80117c
fix(tests): Add missing awaits (#3273)
Also add some more logs to help debug teardown timeout issues
2024-07-11 11:46:28 +03:00
Kostas Kyrimis
9782eb2ac4
fix(acl): loading interleaved plain and hashed passwords (#3297)
* fix a bug of rehashing hashed passwords while loading plan/hashed passwords from acl file

Signed-off-by: kostas <kostas@dragonflydb.io>
2024-07-10 12:40:40 +00:00
Borys
21620ef46f
fix: missing logs on pytest failures #3255 (#3272)
* fix: missing logs on pytest failures #3255
2024-07-10 10:58:54 +03:00
adiholden
5c7c21b6c5
fix(bug): crash on takeover and info replication (#3282)
Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-07-08 17:21:12 +03:00
Roman Gershman
fba902d0ac
fix: properly clean tiered state upon flash (#3281)
* fix: properly clean tiered state upon flash

The bug was around io pending entries that have not been properly cleaned during flush.
This PR simplified the logic around tiered storage handling during flush, it always performs the
cleaning in the synchronous part of the command.

In addition, this PR improves error logging in tests if dragonfly process exits with an error.
Finally, a test is added that makes sure pending tiered items are flushed during the flash call.

Fixes #3252
---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-07-08 10:43:11 +03:00
Shahar Mike
4fd6ba68a2
fix: Increase key count to make test more robust (#3286) 2024-07-08 07:10:03 +00:00
Shahar Mike
2f09055260
chore: Disable cluster_mgr_test (#3279) 2024-07-08 09:12:27 +03:00
Shahar Mike
daa7e3708c
fix(test): Verify that save has not finished (#3278)
fix(test): Verify that save has not finished, otherwise it's an endless loop
2024-07-08 08:33:44 +03:00
Vladislav
2bf4451ec7
fix(tiering): Throttle snapshot load (#3249) 2024-07-04 17:10:23 +03:00
Roman Gershman
d8946247df
chore: upload all the logs from /tmp (#3263)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-07-04 09:17:25 +03:00
Borys
84814a7358
fix: fix move error during migration finalization (#3253)
* fix: fix Move error during migration finalization
2024-07-02 14:23:54 +03:00
Kostas Kyrimis
5956275818
chore: replace session wide fixtures with scope (#3251)
* chore: replace session wide fixtures with scope
2024-07-02 10:26:26 +03:00
Roman Gershman
0e375295e8
chore: introduce back-pressure to tiered storage (#3243)
* chore: introduce back-pressure to tiered storage

Also, so clean-up with mac-os daily build.


Enabled forgotten test.
Improve CI insights

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-06-30 15:46:11 +03:00
Vladislav
4cc9834d89
fix(pytest): timed ticker for simpler conditions (#3242)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-06-29 15:48:25 +03:00
Roman Gershman
b34d3ba6da
chore(tiered): minor fixes + expose buffer allocation type stats (#3234)
* chore(tiered): minor fixes + expose buffer allocation type stats

fixes small_bins deletion logic

Fixes #3240
---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-06-29 11:42:17 +03:00
Vladislav
c1bde661fb
chore: disable tiering test (#3236)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-06-28 10:01:42 +03:00
Kostas Kyrimis
847e2edc09
fix: replicaof_reject_on_load flake (#3203) 2024-06-25 13:52:17 +03:00
Vladislav
4357933775
feat(server): expiry notifications (#3154)
Adds basic support for keyspace notifications, only Ex

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-06-24 16:23:40 +03:00
Vladislav
cf9ac35596
chore: fix tiering regtest test (#3208) 2024-06-23 12:02:46 +03:00
Kostas Kyrimis
e097248baf
chore: add replica-priority flag (#3204)
* add replica-priority flag
* add it on info replication command
* add test
2024-06-22 00:34:48 +03:00
Vladislav
40ede6f61a
chore(tiering): More advanced tiering tests (#3201)
* chore(tiering): More advanced tiering tests

* fix: fixes
2024-06-21 23:47:59 +03:00
Kostas Kyrimis
f01aa2d76b
feat(acl): add support of multiple passwords (#3189)
* add support for multiple passwords
* add support for deleting passwords
* add support for resetpass
* add tests
* always prefix passwords with hashtag when printed
2024-06-21 20:39:33 +03:00
Shahar Mike
48c6f4bf74
chore: Re-enable previously flaky test (#3196) 2024-06-21 13:12:14 +03:00
Shahar Mike
6024d79bd6
feat(cluster): Support STICK bit in slot migration (#3200) 2024-06-21 08:18:03 +03:00
Shahar Mike
c8f2f253d6
test(cluster): Make sure migration maintains TTL (#3188) 2024-06-20 20:46:38 +03:00
Borys
4e7f6dc6ed
test: improve cluster_fuzzy_migration test (#3197) 2024-06-20 19:09:15 +03:00
Shahar Mike
43d7993cda
fix(server): Rename confusing flag replica_reconnect_on_master_restart (#3193)
That was a misleading name, as the logic was the exact opposite (oops 🤦)

This PR introduces a new name for the same flag: break_replication_on_master_restart

We're keeping the previous flag for now, to make transition easier. We'll remove it in a later Dragonfly version (>= 1.22)

Fixes #3192
2024-06-20 15:09:33 +03:00
Stepan Bagritsevich
0156325590
fix(generic_family): fix RenameGeneric command for non-string data types (#3181)
* fix(generic_family): fix RenameGeneric command for non-string data types

fixes dragonflydb#3107, fixes dragonflydb#3113, fixes dragonflydb#307

Signed-off-by: Stepan Bagritsevich <bagr.stepan@gmail.com>

* refactor: address comments

Signed-off-by: Stepan Bagritsevich <bagr.stepan@gmail.com>

* refactor: address comments 2

Signed-off-by: Stepan Bagritsevich <bagr.stepan@gmail.com>

---------

Signed-off-by: Stepan Bagritsevich <bagr.stepan@gmail.com>
2024-06-20 14:03:05 +04:00
Shahar Mike
f66ee5f47d
fix(cluster): Support FLUSHALL while slot migration is in progress (#3173)
* fix(cluster): Support `FLUSHALL` while slot migration is in progress

Fixes #3132

Also do a small refactor to move cancellation logic into
`RestoreStreamer`.
2024-06-20 11:40:23 +03:00
Kostas Kyrimis
d207789610
chore(ci): run replication tests on arm (#3168)
* combine replication tests and reg tests in one flow
* allow replication tests to run on arm
2024-06-18 16:48:35 +03:00
Kostas Kyrimis
8d82581c96
chore: small acl compat changes (#3177)
* print categories and commands in lower case instead of capital case
* fix a bug of default user inheriting the wrong acl rules on new connections
* move keys position to be after password when printed from an acl command
2024-06-17 09:40:41 +03:00
adiholden
805c024fc2
fix(server): fix bug in replication on cached mode (#3156)
* fix server: fix replication on cached mode

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-06-17 08:37:58 +03:00
Vladislav
c08719117c
feat(json): MSET (#3167)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-06-13 12:33:24 +03:00
Kostas Kyrimis
d2ae0ab75c
fix: acl compatibility (#3147)
* remove acl categories from context and all acl checks
* category assign,ent now assigns all the acl commands for that category to the user
* introduce modification order of acl's per user
* acl rules are now printed in the same order as in redis/valkey
* remove old user_registry_test which was part of the poc
2024-06-13 10:56:30 +03:00
Roman Gershman
007d4854db
chore: Introduce pipeline back-pressure (#3152)
* chore: Introduce pipeline back-pressure

Also, improve synchronization primitives and replace them with
thread-local variations.

Before the change, on my local machine with the dragonfly running with 8 threads,
`memtier_benchmark  -c 10 --threads 8  --command="PING"  --key-maximum 100000000  --hide-histogram --distinct-client-seed --pipeline=20 --test-time=10`

reached 10M qps with 0.327ms p99.9.

After the change, the same command showed 13.8M qps with 0.2ms p99.9
---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-06-10 12:39:41 +03:00
Vladislav
1053633a6a
feat(search): Tag field options (separator, case sensitivity) (#3144)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-06-07 22:34:06 +03:00
Borys
39dd73fc71
fix: fix bug in cluster/slot_set (#3143)
* fix: fix bug in cluster/slot_set

* fix: fix slot flushes
2024-06-07 14:31:11 +03:00
Borys
66a524a026
test: skip test_cluster_migration_cancel, it is broken (#3146) 2024-06-06 16:33:08 +03:00
Shahar Mike
229eeeb014
fix: Fix live-lock in connection test (#3135)
fix: Fix livelocking connection test
2024-06-05 21:30:02 +03:00
Shahar Mike
1fb250b64f
test(cluster_mgr): Add tests for cluster_mgr.py (#3129) 2024-06-05 12:12:58 +03:00
Vladislav
6a873b4f1c
feat(tiering): Simple snapshotting (#3073)
* feat(tiering): Simple snapshotting

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-06-04 17:15:21 +03:00
Roman Gershman
b02521cf51
chore: prevent Dispatch fiber to be launched during migration (#3123)
* chore: prevent Dispatch fiber to be launched during the connection migration
2024-06-04 14:13:48 +00:00
Borys
7606af706f
fix: fix RestoreStreamer to prevent buckets skipping #2830 (#3119)
* fix: fix RestoreStreamer to prevent bucket skipping #2830
2024-06-04 11:50:03 +03:00
adiholden
6e33261402
fix(server): fix compatibility with rdb snapshot (#3121)
* fix server: fix compatibility with rdb snapshot


Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-06-04 06:28:18 +00:00
Kostas Kyrimis
3924fcad68
chore: pull helio add test for tls deadlock (#3111)
* pull helio
* add test that covers tls deadlock
2024-06-03 14:13:47 +00:00
Borys
644dc3f139
New test for cluster migration: connection issue (#3102)
* test: update test_config_consistency, 
update test_cluster_data_migration, 
new cluster migration test for network issues
2024-06-02 09:16:03 +03:00
Borys
b02a789ebf
fix: add timeout for DFLYMIGRATE ACK to prevent deadlock (#3093)
* fix: add timeout for DFLYMIGRATE ACK to prevent deadlock
2024-05-28 17:41:51 +03:00
Kostas Kyrimis
2391f49e1c
fix(acl): return -NOPERM instead of response error (#3049) 2024-05-27 09:25:54 +03:00
Borys
0dea257f41
fix: fix cluster incorrect keys status (#3083)
* fix: fix cluster incorrect keys status
2024-05-26 15:10:01 +03:00
Roman Gershman
816cec12ed
chore: pull latest helio (#3009)
Also clean-up unused flags.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-05-05 22:43:59 +03:00
Shahar Mike
082aba02ef
fix(cluster-migration): Support cancelling migration right after starting it (#2992)
* fix(cluster-migration): Support cancelling migration right after starting it

This fixes a few small places, but most importantly it does not allow a
migration to start before both the outgoing and incoming side received
the updated config. This solves a few edge cases.

Fixes #2968

* add TODO

* fix test

* gh comments and fixes

* add comment
2024-05-02 15:50:42 +03:00
Roman Gershman
9bda5b1d4b
chore: another preparation commit to get rid of kv_args in transaction (#2996)
This changes Entry::Payload to struct instead of variant.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-05-02 09:59:45 +03:00
Roman Gershman
07d076a658
chore: remove version checks when running our regtests (#2988) 2024-05-01 13:09:06 +03:00
Kostas Kyrimis
39c7cfdf72
fix: allow non hashed passwords when loading users from acl file (#2982)
* allow non hashed passwords when loading from acl file
2024-05-01 09:57:59 +03:00
Kostas Kyrimis
ab269553ac
fix: check return code of process after communicate (#2976)
* check return code of process after terminate
2024-05-01 09:57:37 +03:00
Vladislav
fd5772a186
chore(tiering): Lots of metrics (#2977)
* chore(tiering): Lots of metrics
2024-04-30 22:25:45 +03:00
Borys
415839df79
fix: fix deadlock and slot flush for migration cancel #2968 (#2972)
* fix: fix deadlock and slot flush for migration cancel #2968
2024-04-30 08:44:05 +00:00
Kostas Kyrimis
652d4da932
chore(acl): allow multiple users in acl deluser (#2945)
* add allow multiple users in acl deluser command
2024-04-29 17:03:54 +03:00
Borys
654ec9f1c4
feat: add slot migration error processing (#2957)
* feat: add slot migration error processing
2024-04-29 10:51:23 +03:00
Kostas Kyrimis
bbe6c8579a
fix(pytests): replace proc.wait() with proc.communicate() to avoid deadlocks (#2964)
* replace proc.wait() with proc.communicate() to avoid deadlocks
2024-04-26 13:22:33 +03:00
adiholden
d5cd0ed204
fixes for v1.18.0 (#2956)
* fix server: change table_growth_margin default value

---------

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-04-24 18:21:14 +03:00
Joe Zhou
84aa237ba7
chore(acl): adjust some ACL command responses (#2943)
* change ACL DELUSER, ACL WHOAMI, and some ACL DRYRUN string/integer responses.
* change ACL GETUSER response, when the user does not exist, it should reply (nil).
2024-04-22 18:14:47 +03:00
Vladislav
e78b909b96
feat(tiering): Get, GetSet, Set test (#2921)
Get, GetSet, Set test

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-04-22 13:55:17 +03:00
Shahar Mike
322b2e7ac1
fix(test): Unflake fuzzy cluster migration test (#2927)
* WIP WIP WIP: Test if fuzzy migration test is still flaky

* tune down

* rm ci changes
2024-04-19 23:04:01 +03:00
Borys
9a6a9ec198
feat: add ability reaply config with migration #2924 (#2926)
* feat: add ability reaply config with migration #2924
2024-04-19 16:21:54 +03:00
Shahar Mike
56965edbe1
feat(cluster): Migration cancellation support (#2869) 2024-04-17 13:19:31 +03:00
Vladislav
4fe00a071e
chore(tiering): Update Get, Set, Del (#2897)
* chore(tiering): Update Get, Set and Del


---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-04-16 19:20:24 +03:00
Borys
d99b0eda16
feat: retry ACK if the configs are different #2833 (#2906)
* feat: retry ACK if the configs are different #2833
2024-04-16 15:03:30 +03:00
Vladislav
468942ccbb
chore: pull helio and add ipv6 replication test (#2889)
* chore: pull helio and add ipv6 replication test

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-04-15 15:37:22 +03:00
adiholden
9cbe69576e
fix(cluster_replication): replicate redis cluster node bug fix (#2876)
* fix redis replication error handling and set cntx as journal emulated


Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-04-14 22:49:00 +03:00
Kostas Kyrimis
c2f13993d9
fix(acl): authentication with UDS socket (#2895)
* disable authentication on UDS socket
* add a test so the bug won't happen again
2024-04-12 16:01:12 +03:00
Roman Gershman
2a4d3ae08e
chore: clean up REPLTAKEOVER flow (#2887)
* chore: clean up REPLTAKEOVER flow

1. Factor out the catchup function.
2. Simplify the flow and make the second parameters - integer.
3. Return OK if the server is already a master (and do nothing underneath).

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-04-11 16:18:06 +03:00
Kostas Kyrimis
1bcbb41ec3
chore: relax repltakeover constraints to only exclude write commands (#2873)
* relax repltakeover constraints
2024-04-10 16:58:34 +03:00
adiholden
eb164be596
fix(redis replication): remove partial sync flow ,not supported yet (#2865)
* fix redis replicaiton: remove partial sync flow ,not supported yet

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-04-10 09:27:02 +03:00
adiholden
b1e688b33f
bug(server): set connection flags block/pause flag on all blocking commands (#2816)
* bug((server)): set connecttion blocking and puash flags on all blocking commands

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-04-09 09:49:33 +03:00
Shahar Mike
b8693b4805
feat(cluster): Send number of keys for incoming and outgoing migrations. (#2858)
The number of keys in an _incoming_ migration indicates how many keys
were received, while for _outgoing_ it shows the total number. Combining
the two can provide the control plane with percentage.

This slightly modified the format of the response.

Fixes #2756
2024-04-08 21:17:03 +03:00
Roman Gershman
604e9c6e97
fix: authorize the http connection to call commands (#2863)
fix: authorize the http connection to call DF commands

The assumption is that basic-auth already covers the authentication part.
And thanks to @sunneydev for finding the bug and providing the tests.
The tests actually uncovered another bug where we may parse partial http requests.
This one is handled by https://github.com/romange/helio/pull/243

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-04-08 13:19:01 +03:00
adiholden
ee8e5a53bf
fix(pytest): dont check process return code on kill (#2862)
fix pytest: dont check process return code on kill

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-04-08 07:08:05 +00:00
adiholden
a5ea47f2d9
fix(pytest): make pytests fail if server crash on shutdown (#2827)
* fix pytests: make pytests fail if server crash on shutdown

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-04-07 15:30:04 +03:00
Vladislav
76729d6e4c
fix(tests): Fix numsub test (#2852)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-04-07 09:48:57 +03:00
Borys
482bd58787
feat(cluster): add migration removing by config #2835 (#2844) 2024-04-05 11:03:54 +03:00
Borys
7b419c6d10
refactor(cluster): replace sync_id with node_id for slot migration #2835 (#2838) 2024-04-04 10:14:03 +03:00
Roman Gershman
8937e28bc2
chore: Fix build by disabling the tests. (#2821)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-04-02 15:21:28 +03:00
Roman Gershman
d3b90c8210
fix: correct json response for errors (#2813)
Fixes #2811

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-04-01 22:56:26 +03:00
Kostas Kyrimis
b2e2ad6e04
feat(server): check master journal lsn in replica (#2778)
Send journal lsn to replica and compare the lsn value against number of records received in replica side

Signed-off-by: kostas <kostas@dragonflydb.io>
Co-authored-by: adi_holden <adi@dragonflydb.io>
2024-04-01 17:51:31 +03:00
Kostas Kyrimis
095ed161d2
chore: disable test_cluster_slot_migration (#2788)
* disable test_cluster_slot_migration because it needs refactoring -- design has changed
2024-03-29 09:51:24 +00:00
Vladislav
c8724adddf
chore: Fix memcached flags not updated (#2787)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-03-29 12:10:58 +03:00
Kostas Kyrimis
cd20c4003d
chore(replication-tests): add cache_mode on test replication all (#2685)
* add cache_mode cases on test_replication_all
* fix CVCOnBumpUp to not skip some of the modified buckets
2024-03-27 14:28:52 +02:00
Kostas Kyrimis
7b7291c001
chore(ci): open last_log_file in append mode (#2776)
* open last_log_file in append mode
2024-03-26 19:01:26 +02:00
adiholden
2ad7439128
feat(server): support cluster replication (#2748)
* feat(server): support cluster replication

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-03-26 15:26:19 +02:00
Kostas Kyrimis
3abee8a361
fix(ci): do not overwrite last_log_file among tests (#2759)
* add clean up semantics for tests such that we don't loose log files
2024-03-26 14:24:02 +02:00
Roman Gershman
966d7f55ba
chore: preparation for basic http api (#2764)
* chore: preparation for basic http api

The goal is to provide very basic support for simple commands,
fancy stuff like pipelining, blocking commands won't work.

1. Added optional registration for /api handler.
2. Implemented parsing of post body.
3. Added basic formatting routine for the response. It does not cover all the commands but should suffice for
   basic usage.

The API is a POST method and the body of the request should contain command arguments formatted as json array.
For example, `'["set", "foo", "bar", "ex", "100"]'`.
The response is a json object with either `result` field holding the response of the command or
`error` field containing the error message sent by the server.
See `test_http` test in tests/dragonfly/connection_test.py for more details.


* chore: cover iouring with enable_direct_fd

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-03-25 12:12:31 +02:00
Shahar Mike
b830a71e66
feat(replication): Do not auto replicate different master (#2753)
* feat(replication): Do not auto replicate different master

Until now, replicas would re-connect and re-replicate a master after the
master will restart. This is problematic in case the master loses its
data, which will cause the replica to flush all and lose its data as
well.

This is a breaking change though, in that whoever controls the replica
now has to explicitly issue a `REPLICAOF X Y` in order to re-establish
a connection to a new master. This is true even if the master loaded an
up to date RDB file.

It's not necessary if the replica lost connection to the master and the
master was always alive, and the connection is re-established.

Fixes #2636

* fix test

* fixes

* proxy proxy java java

* better comment

* fix comments

* replica_reconnect_on_master_restart

* proxy.close()
2024-03-24 12:04:37 +02:00
Borys
d6b7df94bb
refactor: remove start-slot-migration cmd #2727 (#2728)
* refactor: remove start-slot-migration cmd #2727
2024-03-21 13:37:05 +02:00
Roman Gershman
2d246adbbb
chore: better error reporting when connecting to tls with plain socket (#2740)
* chore: better error reporting when connecting to tls with plain socket

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-03-19 17:20:23 +02:00
Shahar Mike
5c69f5552c
fix(cluster): Save replica ID per replica (#2735)
Fixes #2734
2024-03-19 13:42:48 +02:00
Kostas Kyrimis
76d697aaff
feat: add flag masteruser (#2693)
* add flag masteruser
* fix ack reply on acl validation failure
2024-03-15 08:41:24 +02:00
Kostas Kyrimis
094df3ef3f
fix(BgSave): async from sync (#2702)
* make BgSave command async
2024-03-14 13:31:58 +02:00
Kostas Kyrimis
a96d4e4dc8
chore: disable flaky fuzzy migration test (#2716) 2024-03-12 14:18:43 +02:00
Shahar Mike
8b31195798
feat(cluster): Add --cluster_id flag (#2695)
* feat(cluster): Add `--cluster_id` flag

This flag sets the unique ID of a node in a cluster.

It is UB (and bad) to set the same IDs to multiple nodes in the same
cluster.

If unset (default), the `master_replid` (previously known as `master_id`) is used.

Fixes #2643
Related to #2636

* gh comments

* oops - revert line removal

* fix

* replica

* disallow cluster_node_id in emulated mode

* fix replica test
2024-03-10 20:16:31 +02:00
Borys
98616755c0
feat(cluster): automatic slot migration finalization #2697 (#2698)
* feat(cluster): automatic slot migration finalization #2697
2024-03-07 20:29:21 +02:00
Borys
dfedaf7e6e
refactor: remove FULL-SYNC-CUT cmd #2687 (#2688)
* refactor: remove FULL-SYNC-CUT cmd #2687
2024-03-06 14:26:35 +02:00
Roman Gershman
93debc754c
chore: more fixes for macos (#2677)
1.Add back the search files to MacOs build (linker errors are fixed now).
2. Add default maxmemory argument (if not present already) when launching dragonfly process in regression tests.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-03-02 23:14:33 +02:00
Borys
e57067d2fb
refactor(cluster): #2652 initiate migration process from CONFIG cmd (#2667)
* refactor(cluster): #2652 initiate migration process from CONFIG cmd
2024-02-29 16:08:53 +02:00
adiholden
7e4527098b
fix(server): client pause work while blocking commands run (#2584)
fix #2576
fix #2661

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-02-28 11:07:03 +00:00
Vladislav
5ac5e3bdac
fix: replace localhost with 127.0.0.1 in cluster tests (#2665) 2024-02-27 12:53:54 +03:00
Roman Gershman
91c299b33e
chore: Del and NUMINCRBY use json::Path (#2655)
* chore: Del and NUMINCRBY use json::Path

Also, fix various protocol bugs when we sent simple string
instead of sending bulk strings.

Fixed a typo in path.cc that lead to a data race bug.

Finally, flip the flag in regression tests to start covering json::Path code
and added test coverage for the data race bug

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-26 14:06:21 +00:00
Vladislav
5ee61db0f3
feat(connection): Support pipelining with Memcached (#2648)
* feat(connection): Support pipelining with Memcached

Adds support for pipelining to Memcached, enhances Memcached pytests

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-23 20:18:25 +03:00
adiholden
9e66ec5833
fix(bug): server crash info memory while saving (#2637)
* bug: crash info memory while saving

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-02-21 13:39:57 +00:00
Kostas Kyrimis
f32156788e
feat(regTests): upload only failed test logs on ci and clean up logging (#2547)
* upload only failed test logs
* remove printing log names for passed tests
* print slow tests with --duration
* separate regression and unit logs for CI workflow
2024-02-21 10:35:07 +02:00
Kostas Kyrimis
a1950038b6
fix: loading state error type to be compatible with redis (#2629)
* add -LOADING prefix for loading errors
* replace -ERR with -LOADING for loading errors
2024-02-20 12:45:41 +00:00
Vladislav
75eaeb32db
feat(pytest): More types for seeder (#2577)
* feat(pytest): More types for seeder

Add more types to the seeder and refactor replication test

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-20 15:31:08 +03:00
adiholden
1ef8795611
fix(bug): crash when starting replica while saving (#2618)
The bug: crash when starting replica while saving
The problem: accessing the wrong allocator on snapshot class destruction as it was destructed not in the thread of the shard
The fix: call snapshot destructor when we finish snapshot on the correct thread 

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-02-20 10:50:46 +00:00
Borys
491538e170
feat(cluster): #2448 add new DFLYMIGRATE ACK cmd (#2582)
* feat(cluster): #2448 add new DFLYMIGRATE ACK cmd
2024-02-19 19:21:14 +02:00
Leonardo Mello
07a8411ee7
fix(server_family): GetMetrics should show commands in lowercase (#2601)
* replace with lowercase cmdstat_info and cmdstat_replconf

Signed-off-by: Leonardo Mello <lsvmello@gmail.com>
2024-02-19 18:50:59 +02:00
adiholden
15b3fb13b6
fix(server): saving is not a server state (#2613)
* fix(server): saving is not a server state

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-02-19 15:20:48 +00:00
Vladislav
d035111bed
fix: fix cluster test (#2612)
* fix: fix cluster test
2024-02-19 11:03:04 +03:00
Shahar Mike
28800df071
fix(test): Use less memory for STRING and HASH memory tests (#2593)
While at it, also register the opt_only mark
2024-02-15 10:08:45 +02:00
Shahar Mike
b18fe8c0a8
test(cluster): Fix and uncomment counter sanity checks (#2591) 2024-02-14 14:06:34 +02:00
Vladislav
4e3be726c5
chore(pytest): Refactor snapshot test (#2583)
* chore(pytest): Refactor snapshot test

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-14 12:13:52 +03:00
Roman Gershman
4000adf57f
fix: do not migrate during connection close (#2570)
* fix: do not migrate during connection close

Fixes #2569
Before the change we had a corner case where Dragonfly would call
OnPreMigrateThread but would not call CancelOnErrorCb because OnBreakCb has already been called
(it resets break_cb_engaged_)

On the other hand in OnPostMigrateThread we called RegisterOnErrorCb if breaker_cb_ which resulted in double registration.
This change simplifies the logic by removing break_cb_engaged_ flag since CancelOnErrorCb is safe to call if nothing is registered.
Moreover, we now skip Migrate flow if a socket is being closed.

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-12 16:03:34 +02:00
Shahar Mike
6d11f86091
test(cluster-migration): Fix some bugs and add cluster migration fuzzy tests (#2572)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
Co-authored-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-12 13:47:34 +02:00
Roman Gershman
24fcf8d883
chore: Recognize exit status in regression tests (#2571)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-12 09:03:56 +02:00
Shahar Mike
8ead569b2f
test(memory): Unaccounted memory test + add DEBUG POPULATE TYPE <type> (#2561)
* test(memory): Test memory accounting for all types

* slightly faster

* WIP

* working

* Document

* Update test to use DEBUG POPULATE

* Nothing much

* Working

* fix

* yaml

* explicit capture

* fix ci?

* stub tx
2024-02-12 08:09:48 +02:00
Vladislav
881edb501e
feat(pytest): Gen2 seeder, part 1 (#2556)
* feat(pytest): Gen2 seeder

Implement new seeder that uses lua scripts to improve performance

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-09 19:20:25 +03:00
Vladislav
9537cbdb0b
fix(pytest): Disable flaky interpreter test monitoring (#2558) 2024-02-07 12:51:24 +03:00
Vladislav
ed59a439d1
fix(tests): increase interpreter test load, lower notice limit, include 2553 (#2554)
* fix: increase load + lower notice limit

* chore: include 2553

* fix: lower metric EVEN more

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-06 16:57:32 +02:00
Vladislav
83a12b99c6
fix: fix interpreter acquisition with MULTI (#2549)
* fix: fix interpreter acquisition with MULTI

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-06 12:54:14 +02:00
Shahar Mike
bc9b214ae4
fix(server): Do not yield in journal if not allowed (#2540)
* fix(server): Do not yield in journal if not allowed

* Add pytest

* Compare keys

* check_all_replicas_finished
2024-02-06 12:35:00 +02:00
Kostas Kyrimis
7b4591cc7d
fix: wrongly printing timedout emoji on test failures (#2546)
* fix wrongly printing timedout emoji on non timedout test failures
* add pytest-timeout as dependency in tests/dragonfly/requirements
2024-02-06 10:25:12 +02:00
Borys
eaca66b02f
fix(cluster): fix #2532 tests redis client close errors (#2542)
* fix(cluster): fix #2532 tests redis client close errors

* refactor: address comments
2024-02-05 22:42:03 +02:00
Roman Gershman
5c0029978e
chore: remove redis sorted set implementation (#2522)
Also remove unused code.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-05 10:29:11 +02:00
Vladislav
e0f86697f9
fix: fix script replication (#2531)
* fix: fix script replication

Single key script replication was previously broken because the EXEC entry wasn't sent. Send it manually

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-04 20:28:44 +03:00
Borys
5189dae118
feat(cluster): add migration finalization (#2507)
* feat(cluster): add migration finalization
2024-02-01 17:24:54 +02:00
Shahar Mike
ad90602bc2
test: Disable flaky test test_policy_based_eviction_propagation (#2505) 2024-01-30 10:28:00 +02:00
Roman Gershman
97bde35a0f
Fix redis->DF replication tests (#2495)
fix: redis_replication_test fails due to long time for df replicas takes to catch up.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-01-29 03:25:58 +02:00
Kostas Kyrimis
754a186f5a
feat(lua): add missing no-op redis.replicate_commands() (#2472)
fixes #2468
add no-op lua function redis.replicate_commands()
2024-01-28 13:05:52 +02:00
Borys
43808da27f
fix(cluster): fix slot filtration to RestoreStreamer (#2477)
* fix(cluster): fix slot filtration to RestoreStreamer

* test: add cluster data migration test
2024-01-28 12:29:54 +02:00
Vladislav
a5b9401449
fix: reduce test_pipeline_batching_while_migrating flakiness (#2475)
* fix: reduce test_pipeline_batching_while_migrating flakiness
2024-01-25 17:55:12 +03:00
Kostas Kyrimis
ba972923b3
feat(lua): add no-op redis.log command (#2476)
* add no-op redis.log command
2024-01-25 15:45:47 +02:00
Shahar Mike
e6f418575b
test(cluster): Enable seeder to work against a Dragonfly cluster (#2462) 2024-01-24 20:02:04 +02:00
Borys
a16b940a65
feat(cluster): add tx execution in cluster_shard_migration (#2385)
* feat(cluster): add tx execution in cluster_shard_migration
refactor(replication): move code that is common for cluster and
replica into a separate file, add full-sync-cut cmd
2024-01-22 21:19:39 +02:00
Shahar Mike
2f0287429d
fix(replication): Correctly replicate commands even when OOM (#2428)
* fix(replication): Correctly replicate commands even when OOM

Before this change, OOM in shard callbacks could have led to data
inconsistency between the master and the replica. For example, commands
which mutated data on 1 shard but failed on another, like `LMOVE`.

After this change, callbacks that result in an OOM will correctly
replicate their work (none, partial or complete) to replicas.

Note that `MSET` and `MSETNX` required special handling, in that they are
the only commands that can _create_ multiple keys, and so some of them
can fail.

Fixes #2381

* fixes

* test fix

* RecordJournal

* UNDO idiotnessness

* 2 shards

* fix pytest
2024-01-18 12:29:59 +02:00
Kostas Kyrimis
39e7e5ad87
fix: missing error reply to client after AddOrFind throw std::bad_alloc (#2411)
* Handle properly and reply on execution paths that throw std::bad_alloc within AddOrFind
2024-01-15 10:13:10 +02:00
Shahar Mike
13718699d8
feat(server): Implement CLIENT KILL (#2404)
* feat(server): Implement `CLIENT KILL`

Currently, it supports the following syntax:

* `CLIENT KILL <addr>:<port>`
* `CLIENT KILL ID <id>`
* `CLIENT KILL ADDR <addr>:<port>`
* `CLIENT KILL LADDR <addr>:<port>`

It will not allow killing an admin-connection from a non-admin port.

There are a few parameters of `CLIENT KILL` that Redis supports but this
PR does not yet add. Let's add them as needed.

Fixes #1614

* Add tests

* fixes
2024-01-15 09:49:23 +02:00
Vladislav
484b4de216
Fix flush when migrating connection (#2407)
fix: don't miss flush for control messages
2024-01-13 09:57:33 +03:00
Yue Li
8d09478474
bug(server): log evicted keys in journal in PrimeEvictionPolicy. (#2302)
fixes #2296

added a regression test that tests both policy based eviction as well as heart beat eviction.

---------

Signed-off-by: Yue Li <61070669+theyueli@users.noreply.github.com>
2024-01-11 01:45:29 -08:00
adiholden
f37c57c704
fix(server): crash on rename save command on background save (#2375)
* fix(server): crash on rename save command on baground save

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-01-07 12:21:09 +02:00
Yue Li
6f9107291e
test: Adding integration test using Relay benchmark (#2348)
Adding integration test using Relay benchmark
2024-01-02 12:44:22 -08:00
Borys
03f69ff6c3
feat: add SLOT-MIGRATION-STATUS cmd for source node (#2349)
* feat: add SLOT-MIGRATION-STATUS cmd for source node
implements #2232
add ability using SLOT-MIGRATION-STATUS without args
to print info about all migration processes for the current node
2024-01-02 12:10:06 +02:00
adiholden
5d67c95797
bug(server): reject replicaof while loading from snapshot (#2338)
fix #2337
The bug:
replicaof was not rejected while loading snapshot
The fix:
replicaof is allowed while server is in loading state to allow replicaof while replication in full sync mode
I now reject replicaof if the server is in loading state and it is master

Another bug fix:
allow cron snapshot if --replicaof flag was set

Signed-off-by: adi_holden <adi@dragonflydb.io>
2023-12-27 13:57:49 +02:00
Shahar Mike
a360b308c9
refactor(server): Privatize PreUpdate() and PostUpdate() (#2322)
* refactor(server): Privatize `PreUpdate()` and `PostUpdate()`

While at it:
* Make `PreUpdate()` not decrease object size
* Remove redundant leftover call to `PreUpdate()` outside `DbSlice`

* Add pytest

* Test delete leads to 0 counters

* Improve test

* fixes

* comments
2023-12-25 07:49:57 +00:00
Roman Gershman
700a65ece5
chore: refactor VersionMonitor into a separate file (#2326)
* chore: refactor VersionMonitor into a separate file
---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-12-24 22:06:57 +02:00
Roman Gershman
bbe3d9303b
feat: introduce transaction statistics in the info output (#2328)
1. How many transactions we processed by type
2. How many transactions we processed by width (number of unique shards).

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-12-23 13:18:49 +02:00
Roman Gershman
365cb439cf
chore: remove support for save_schedule flag (#2327)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-12-22 11:17:18 +02:00
Borys
fd76c51310
feat: add command flow for slot migration process (#2292)
* feat(cluster): add command flow for slot migration process
fixes #2295

DFLYMIGRATE FLOW command was added to establish
connections for every shard replication process.
Slow serialization step is the separate issue so
for now only eof_token is sent for reply to
DFLYMIGRATE FLOW command.
Expected state for START-SLOT-MIGRATION is FULL_SYNC now.
2023-12-20 18:47:11 +02:00
Vladislav
aaf01d4244
feat(cluster): Cancel blocking commands on cluster update (#2255)
Handle blocking commands during cluster config update

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-12-17 15:32:35 +03:00
s-shiraki
bd3e57d262
feat(server): Implement NUMSUB subcommand (#2282)
* feat(server): Implement NUMSUB subcommand

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix: test

* fix: build error
2023-12-16 20:42:15 +02:00
Vladislav
7ca07a498f
fix(server): Fix client pause and add test (#2298)
Fixes a bug in which we incorrectly determined paused dispatches, which led to not allowing multiple (overlapping) client pauses
2023-12-12 19:28:48 +03:00
Kostas Kyrimis
8640edad71
feat(acl): add acl keys to acl log command (#2274)
* add acl keys to acl log command
* add tests
2023-12-12 17:00:41 +02:00
Kostas Kyrimis
8323c82dc5
feat(acl): add acl keys to acl save/load (#2273)
* add acl keys to acl savel/load
* add tests
2023-12-08 16:08:33 +00:00
Kostas Kyrimis
2703d4635d
feat(acl): add validation for acl keys (#2272)
* add validation for acl keys
* add tests
2023-12-08 17:28:53 +02:00
Kostas Kyrimis
8126cf8252
feat(acl): add acl keys to acl list command (#2261)
* add acl keys to acl list
2023-12-08 15:32:15 +03:00
Vladislav
11ef6623dc
feat: DispatchTracker to replace everything (#2179)
* feat: DispatchTracker

Use a DispatchTracker to track ongoing dispatches for commands that change global state

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-12-05 11:02:11 +03:00
Borys
24b13434cf
feat: add slot-migration-status command (#2239)
* feat: add slot-migration-status command
2023-12-04 12:47:46 +02:00
Roman Gershman
26512fdba4
fix: remove string copy in SendMGetResponse (#2246)
fix: eliminate the redundant string copy in SendMGetResponse

Also, allow selectively create DflyInstance in pytests that is attached to
an existing dragonfly port, created outside of tests.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-12-03 18:14:19 +02:00
Borys
bfb1b3b624
Start slot migration (#2218)
* feat: add new command START-SLOT-MIGRATION
2023-11-29 13:38:13 +02:00
Roman Gershman
b853b2ab00
fix: memcached VERSION is now parseable by php-memcached client (#2220)
The DF version is being unparseable by Memcached::getVersion() that expects n.n.n string.
Change the version to emulate the old memcached server.
The DF version can still be fetched via Memcached::getStats() function.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-11-27 20:54:00 +02:00