1
0
Fork 0
mirror of https://github.com/dragonflydb/dragonfly.git synced 2024-12-14 11:58:02 +00:00
Commit graph

567 commits

Author SHA1 Message Date
adiholden
b1e688b33f
bug(server): set connection flags block/pause flag on all blocking commands (#2816)
* bug((server)): set connecttion blocking and puash flags on all blocking commands

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-04-09 09:49:33 +03:00
Shahar Mike
b8693b4805
feat(cluster): Send number of keys for incoming and outgoing migrations. (#2858)
The number of keys in an _incoming_ migration indicates how many keys
were received, while for _outgoing_ it shows the total number. Combining
the two can provide the control plane with percentage.

This slightly modified the format of the response.

Fixes #2756
2024-04-08 21:17:03 +03:00
Roman Gershman
604e9c6e97
fix: authorize the http connection to call commands (#2863)
fix: authorize the http connection to call DF commands

The assumption is that basic-auth already covers the authentication part.
And thanks to @sunneydev for finding the bug and providing the tests.
The tests actually uncovered another bug where we may parse partial http requests.
This one is handled by https://github.com/romange/helio/pull/243

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-04-08 13:19:01 +03:00
adiholden
ee8e5a53bf
fix(pytest): dont check process return code on kill (#2862)
fix pytest: dont check process return code on kill

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-04-08 07:08:05 +00:00
adiholden
a5ea47f2d9
fix(pytest): make pytests fail if server crash on shutdown (#2827)
* fix pytests: make pytests fail if server crash on shutdown

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-04-07 15:30:04 +03:00
Vladislav
76729d6e4c
fix(tests): Fix numsub test (#2852)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-04-07 09:48:57 +03:00
Borys
482bd58787
feat(cluster): add migration removing by config #2835 (#2844) 2024-04-05 11:03:54 +03:00
Borys
7b419c6d10
refactor(cluster): replace sync_id with node_id for slot migration #2835 (#2838) 2024-04-04 10:14:03 +03:00
Roman Gershman
8937e28bc2
chore: Fix build by disabling the tests. (#2821)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-04-02 15:21:28 +03:00
Roman Gershman
d3b90c8210
fix: correct json response for errors (#2813)
Fixes #2811

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-04-01 22:56:26 +03:00
Kostas Kyrimis
b2e2ad6e04
feat(server): check master journal lsn in replica (#2778)
Send journal lsn to replica and compare the lsn value against number of records received in replica side

Signed-off-by: kostas <kostas@dragonflydb.io>
Co-authored-by: adi_holden <adi@dragonflydb.io>
2024-04-01 17:51:31 +03:00
Kostas Kyrimis
095ed161d2
chore: disable test_cluster_slot_migration (#2788)
* disable test_cluster_slot_migration because it needs refactoring -- design has changed
2024-03-29 09:51:24 +00:00
Vladislav
c8724adddf
chore: Fix memcached flags not updated (#2787)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-03-29 12:10:58 +03:00
Kostas Kyrimis
cd20c4003d
chore(replication-tests): add cache_mode on test replication all (#2685)
* add cache_mode cases on test_replication_all
* fix CVCOnBumpUp to not skip some of the modified buckets
2024-03-27 14:28:52 +02:00
Kostas Kyrimis
7b7291c001
chore(ci): open last_log_file in append mode (#2776)
* open last_log_file in append mode
2024-03-26 19:01:26 +02:00
adiholden
2ad7439128
feat(server): support cluster replication (#2748)
* feat(server): support cluster replication

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-03-26 15:26:19 +02:00
Kostas Kyrimis
3abee8a361
fix(ci): do not overwrite last_log_file among tests (#2759)
* add clean up semantics for tests such that we don't loose log files
2024-03-26 14:24:02 +02:00
Roman Gershman
966d7f55ba
chore: preparation for basic http api (#2764)
* chore: preparation for basic http api

The goal is to provide very basic support for simple commands,
fancy stuff like pipelining, blocking commands won't work.

1. Added optional registration for /api handler.
2. Implemented parsing of post body.
3. Added basic formatting routine for the response. It does not cover all the commands but should suffice for
   basic usage.

The API is a POST method and the body of the request should contain command arguments formatted as json array.
For example, `'["set", "foo", "bar", "ex", "100"]'`.
The response is a json object with either `result` field holding the response of the command or
`error` field containing the error message sent by the server.
See `test_http` test in tests/dragonfly/connection_test.py for more details.


* chore: cover iouring with enable_direct_fd

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-03-25 12:12:31 +02:00
Shahar Mike
b830a71e66
feat(replication): Do not auto replicate different master (#2753)
* feat(replication): Do not auto replicate different master

Until now, replicas would re-connect and re-replicate a master after the
master will restart. This is problematic in case the master loses its
data, which will cause the replica to flush all and lose its data as
well.

This is a breaking change though, in that whoever controls the replica
now has to explicitly issue a `REPLICAOF X Y` in order to re-establish
a connection to a new master. This is true even if the master loaded an
up to date RDB file.

It's not necessary if the replica lost connection to the master and the
master was always alive, and the connection is re-established.

Fixes #2636

* fix test

* fixes

* proxy proxy java java

* better comment

* fix comments

* replica_reconnect_on_master_restart

* proxy.close()
2024-03-24 12:04:37 +02:00
Borys
d6b7df94bb
refactor: remove start-slot-migration cmd #2727 (#2728)
* refactor: remove start-slot-migration cmd #2727
2024-03-21 13:37:05 +02:00
Roman Gershman
2d246adbbb
chore: better error reporting when connecting to tls with plain socket (#2740)
* chore: better error reporting when connecting to tls with plain socket

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-03-19 17:20:23 +02:00
Shahar Mike
5c69f5552c
fix(cluster): Save replica ID per replica (#2735)
Fixes #2734
2024-03-19 13:42:48 +02:00
Kostas Kyrimis
76d697aaff
feat: add flag masteruser (#2693)
* add flag masteruser
* fix ack reply on acl validation failure
2024-03-15 08:41:24 +02:00
Kostas Kyrimis
094df3ef3f
fix(BgSave): async from sync (#2702)
* make BgSave command async
2024-03-14 13:31:58 +02:00
Kostas Kyrimis
a96d4e4dc8
chore: disable flaky fuzzy migration test (#2716) 2024-03-12 14:18:43 +02:00
Shahar Mike
8b31195798
feat(cluster): Add --cluster_id flag (#2695)
* feat(cluster): Add `--cluster_id` flag

This flag sets the unique ID of a node in a cluster.

It is UB (and bad) to set the same IDs to multiple nodes in the same
cluster.

If unset (default), the `master_replid` (previously known as `master_id`) is used.

Fixes #2643
Related to #2636

* gh comments

* oops - revert line removal

* fix

* replica

* disallow cluster_node_id in emulated mode

* fix replica test
2024-03-10 20:16:31 +02:00
Borys
98616755c0
feat(cluster): automatic slot migration finalization #2697 (#2698)
* feat(cluster): automatic slot migration finalization #2697
2024-03-07 20:29:21 +02:00
Borys
dfedaf7e6e
refactor: remove FULL-SYNC-CUT cmd #2687 (#2688)
* refactor: remove FULL-SYNC-CUT cmd #2687
2024-03-06 14:26:35 +02:00
Roman Gershman
93debc754c
chore: more fixes for macos (#2677)
1.Add back the search files to MacOs build (linker errors are fixed now).
2. Add default maxmemory argument (if not present already) when launching dragonfly process in regression tests.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-03-02 23:14:33 +02:00
Borys
e57067d2fb
refactor(cluster): #2652 initiate migration process from CONFIG cmd (#2667)
* refactor(cluster): #2652 initiate migration process from CONFIG cmd
2024-02-29 16:08:53 +02:00
adiholden
7e4527098b
fix(server): client pause work while blocking commands run (#2584)
fix #2576
fix #2661

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-02-28 11:07:03 +00:00
Vladislav
5ac5e3bdac
fix: replace localhost with 127.0.0.1 in cluster tests (#2665) 2024-02-27 12:53:54 +03:00
Roman Gershman
91c299b33e
chore: Del and NUMINCRBY use json::Path (#2655)
* chore: Del and NUMINCRBY use json::Path

Also, fix various protocol bugs when we sent simple string
instead of sending bulk strings.

Fixed a typo in path.cc that lead to a data race bug.

Finally, flip the flag in regression tests to start covering json::Path code
and added test coverage for the data race bug

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-26 14:06:21 +00:00
Vladislav
5ee61db0f3
feat(connection): Support pipelining with Memcached (#2648)
* feat(connection): Support pipelining with Memcached

Adds support for pipelining to Memcached, enhances Memcached pytests

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-23 20:18:25 +03:00
adiholden
9e66ec5833
fix(bug): server crash info memory while saving (#2637)
* bug: crash info memory while saving

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-02-21 13:39:57 +00:00
Kostas Kyrimis
f32156788e
feat(regTests): upload only failed test logs on ci and clean up logging (#2547)
* upload only failed test logs
* remove printing log names for passed tests
* print slow tests with --duration
* separate regression and unit logs for CI workflow
2024-02-21 10:35:07 +02:00
Kostas Kyrimis
a1950038b6
fix: loading state error type to be compatible with redis (#2629)
* add -LOADING prefix for loading errors
* replace -ERR with -LOADING for loading errors
2024-02-20 12:45:41 +00:00
Vladislav
75eaeb32db
feat(pytest): More types for seeder (#2577)
* feat(pytest): More types for seeder

Add more types to the seeder and refactor replication test

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-20 15:31:08 +03:00
adiholden
1ef8795611
fix(bug): crash when starting replica while saving (#2618)
The bug: crash when starting replica while saving
The problem: accessing the wrong allocator on snapshot class destruction as it was destructed not in the thread of the shard
The fix: call snapshot destructor when we finish snapshot on the correct thread 

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-02-20 10:50:46 +00:00
Borys
491538e170
feat(cluster): #2448 add new DFLYMIGRATE ACK cmd (#2582)
* feat(cluster): #2448 add new DFLYMIGRATE ACK cmd
2024-02-19 19:21:14 +02:00
Leonardo Mello
07a8411ee7
fix(server_family): GetMetrics should show commands in lowercase (#2601)
* replace with lowercase cmdstat_info and cmdstat_replconf

Signed-off-by: Leonardo Mello <lsvmello@gmail.com>
2024-02-19 18:50:59 +02:00
adiholden
15b3fb13b6
fix(server): saving is not a server state (#2613)
* fix(server): saving is not a server state

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-02-19 15:20:48 +00:00
Vladislav
d035111bed
fix: fix cluster test (#2612)
* fix: fix cluster test
2024-02-19 11:03:04 +03:00
Shahar Mike
28800df071
fix(test): Use less memory for STRING and HASH memory tests (#2593)
While at it, also register the opt_only mark
2024-02-15 10:08:45 +02:00
Shahar Mike
b18fe8c0a8
test(cluster): Fix and uncomment counter sanity checks (#2591) 2024-02-14 14:06:34 +02:00
Vladislav
4e3be726c5
chore(pytest): Refactor snapshot test (#2583)
* chore(pytest): Refactor snapshot test

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-14 12:13:52 +03:00
Roman Gershman
4000adf57f
fix: do not migrate during connection close (#2570)
* fix: do not migrate during connection close

Fixes #2569
Before the change we had a corner case where Dragonfly would call
OnPreMigrateThread but would not call CancelOnErrorCb because OnBreakCb has already been called
(it resets break_cb_engaged_)

On the other hand in OnPostMigrateThread we called RegisterOnErrorCb if breaker_cb_ which resulted in double registration.
This change simplifies the logic by removing break_cb_engaged_ flag since CancelOnErrorCb is safe to call if nothing is registered.
Moreover, we now skip Migrate flow if a socket is being closed.

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-12 16:03:34 +02:00
Shahar Mike
6d11f86091
test(cluster-migration): Fix some bugs and add cluster migration fuzzy tests (#2572)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
Co-authored-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-12 13:47:34 +02:00
Roman Gershman
24fcf8d883
chore: Recognize exit status in regression tests (#2571)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-12 09:03:56 +02:00
Shahar Mike
8ead569b2f
test(memory): Unaccounted memory test + add DEBUG POPULATE TYPE <type> (#2561)
* test(memory): Test memory accounting for all types

* slightly faster

* WIP

* working

* Document

* Update test to use DEBUG POPULATE

* Nothing much

* Working

* fix

* yaml

* explicit capture

* fix ci?

* stub tx
2024-02-12 08:09:48 +02:00
Vladislav
881edb501e
feat(pytest): Gen2 seeder, part 1 (#2556)
* feat(pytest): Gen2 seeder

Implement new seeder that uses lua scripts to improve performance

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-09 19:20:25 +03:00
Vladislav
9537cbdb0b
fix(pytest): Disable flaky interpreter test monitoring (#2558) 2024-02-07 12:51:24 +03:00
Vladislav
ed59a439d1
fix(tests): increase interpreter test load, lower notice limit, include 2553 (#2554)
* fix: increase load + lower notice limit

* chore: include 2553

* fix: lower metric EVEN more

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-06 16:57:32 +02:00
Vladislav
83a12b99c6
fix: fix interpreter acquisition with MULTI (#2549)
* fix: fix interpreter acquisition with MULTI

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-06 12:54:14 +02:00
Shahar Mike
bc9b214ae4
fix(server): Do not yield in journal if not allowed (#2540)
* fix(server): Do not yield in journal if not allowed

* Add pytest

* Compare keys

* check_all_replicas_finished
2024-02-06 12:35:00 +02:00
Kostas Kyrimis
7b4591cc7d
fix: wrongly printing timedout emoji on test failures (#2546)
* fix wrongly printing timedout emoji on non timedout test failures
* add pytest-timeout as dependency in tests/dragonfly/requirements
2024-02-06 10:25:12 +02:00
Borys
eaca66b02f
fix(cluster): fix #2532 tests redis client close errors (#2542)
* fix(cluster): fix #2532 tests redis client close errors

* refactor: address comments
2024-02-05 22:42:03 +02:00
Roman Gershman
5c0029978e
chore: remove redis sorted set implementation (#2522)
Also remove unused code.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-02-05 10:29:11 +02:00
Vladislav
e0f86697f9
fix: fix script replication (#2531)
* fix: fix script replication

Single key script replication was previously broken because the EXEC entry wasn't sent. Send it manually

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2024-02-04 20:28:44 +03:00
Borys
5189dae118
feat(cluster): add migration finalization (#2507)
* feat(cluster): add migration finalization
2024-02-01 17:24:54 +02:00
Shahar Mike
ad90602bc2
test: Disable flaky test test_policy_based_eviction_propagation (#2505) 2024-01-30 10:28:00 +02:00
Roman Gershman
97bde35a0f
Fix redis->DF replication tests (#2495)
fix: redis_replication_test fails due to long time for df replicas takes to catch up.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-01-29 03:25:58 +02:00
Kostas Kyrimis
754a186f5a
feat(lua): add missing no-op redis.replicate_commands() (#2472)
fixes #2468
add no-op lua function redis.replicate_commands()
2024-01-28 13:05:52 +02:00
Borys
43808da27f
fix(cluster): fix slot filtration to RestoreStreamer (#2477)
* fix(cluster): fix slot filtration to RestoreStreamer

* test: add cluster data migration test
2024-01-28 12:29:54 +02:00
Vladislav
a5b9401449
fix: reduce test_pipeline_batching_while_migrating flakiness (#2475)
* fix: reduce test_pipeline_batching_while_migrating flakiness
2024-01-25 17:55:12 +03:00
Kostas Kyrimis
ba972923b3
feat(lua): add no-op redis.log command (#2476)
* add no-op redis.log command
2024-01-25 15:45:47 +02:00
Shahar Mike
e6f418575b
test(cluster): Enable seeder to work against a Dragonfly cluster (#2462) 2024-01-24 20:02:04 +02:00
Borys
a16b940a65
feat(cluster): add tx execution in cluster_shard_migration (#2385)
* feat(cluster): add tx execution in cluster_shard_migration
refactor(replication): move code that is common for cluster and
replica into a separate file, add full-sync-cut cmd
2024-01-22 21:19:39 +02:00
Shahar Mike
2f0287429d
fix(replication): Correctly replicate commands even when OOM (#2428)
* fix(replication): Correctly replicate commands even when OOM

Before this change, OOM in shard callbacks could have led to data
inconsistency between the master and the replica. For example, commands
which mutated data on 1 shard but failed on another, like `LMOVE`.

After this change, callbacks that result in an OOM will correctly
replicate their work (none, partial or complete) to replicas.

Note that `MSET` and `MSETNX` required special handling, in that they are
the only commands that can _create_ multiple keys, and so some of them
can fail.

Fixes #2381

* fixes

* test fix

* RecordJournal

* UNDO idiotnessness

* 2 shards

* fix pytest
2024-01-18 12:29:59 +02:00
Kostas Kyrimis
39e7e5ad87
fix: missing error reply to client after AddOrFind throw std::bad_alloc (#2411)
* Handle properly and reply on execution paths that throw std::bad_alloc within AddOrFind
2024-01-15 10:13:10 +02:00
Shahar Mike
13718699d8
feat(server): Implement CLIENT KILL (#2404)
* feat(server): Implement `CLIENT KILL`

Currently, it supports the following syntax:

* `CLIENT KILL <addr>:<port>`
* `CLIENT KILL ID <id>`
* `CLIENT KILL ADDR <addr>:<port>`
* `CLIENT KILL LADDR <addr>:<port>`

It will not allow killing an admin-connection from a non-admin port.

There are a few parameters of `CLIENT KILL` that Redis supports but this
PR does not yet add. Let's add them as needed.

Fixes #1614

* Add tests

* fixes
2024-01-15 09:49:23 +02:00
Vladislav
484b4de216
Fix flush when migrating connection (#2407)
fix: don't miss flush for control messages
2024-01-13 09:57:33 +03:00
Yue Li
8d09478474
bug(server): log evicted keys in journal in PrimeEvictionPolicy. (#2302)
fixes #2296

added a regression test that tests both policy based eviction as well as heart beat eviction.

---------

Signed-off-by: Yue Li <61070669+theyueli@users.noreply.github.com>
2024-01-11 01:45:29 -08:00
adiholden
f37c57c704
fix(server): crash on rename save command on background save (#2375)
* fix(server): crash on rename save command on baground save

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-01-07 12:21:09 +02:00
Borys
03f69ff6c3
feat: add SLOT-MIGRATION-STATUS cmd for source node (#2349)
* feat: add SLOT-MIGRATION-STATUS cmd for source node
implements #2232
add ability using SLOT-MIGRATION-STATUS without args
to print info about all migration processes for the current node
2024-01-02 12:10:06 +02:00
adiholden
5d67c95797
bug(server): reject replicaof while loading from snapshot (#2338)
fix #2337
The bug:
replicaof was not rejected while loading snapshot
The fix:
replicaof is allowed while server is in loading state to allow replicaof while replication in full sync mode
I now reject replicaof if the server is in loading state and it is master

Another bug fix:
allow cron snapshot if --replicaof flag was set

Signed-off-by: adi_holden <adi@dragonflydb.io>
2023-12-27 13:57:49 +02:00
Shahar Mike
a360b308c9
refactor(server): Privatize PreUpdate() and PostUpdate() (#2322)
* refactor(server): Privatize `PreUpdate()` and `PostUpdate()`

While at it:
* Make `PreUpdate()` not decrease object size
* Remove redundant leftover call to `PreUpdate()` outside `DbSlice`

* Add pytest

* Test delete leads to 0 counters

* Improve test

* fixes

* comments
2023-12-25 07:49:57 +00:00
Roman Gershman
700a65ece5
chore: refactor VersionMonitor into a separate file (#2326)
* chore: refactor VersionMonitor into a separate file
---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-12-24 22:06:57 +02:00
Roman Gershman
bbe3d9303b
feat: introduce transaction statistics in the info output (#2328)
1. How many transactions we processed by type
2. How many transactions we processed by width (number of unique shards).

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-12-23 13:18:49 +02:00
Roman Gershman
365cb439cf
chore: remove support for save_schedule flag (#2327)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-12-22 11:17:18 +02:00
Borys
fd76c51310
feat: add command flow for slot migration process (#2292)
* feat(cluster): add command flow for slot migration process
fixes #2295

DFLYMIGRATE FLOW command was added to establish
connections for every shard replication process.
Slow serialization step is the separate issue so
for now only eof_token is sent for reply to
DFLYMIGRATE FLOW command.
Expected state for START-SLOT-MIGRATION is FULL_SYNC now.
2023-12-20 18:47:11 +02:00
Vladislav
aaf01d4244
feat(cluster): Cancel blocking commands on cluster update (#2255)
Handle blocking commands during cluster config update

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-12-17 15:32:35 +03:00
s-shiraki
bd3e57d262
feat(server): Implement NUMSUB subcommand (#2282)
* feat(server): Implement NUMSUB subcommand

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix: test

* fix: build error
2023-12-16 20:42:15 +02:00
Vladislav
7ca07a498f
fix(server): Fix client pause and add test (#2298)
Fixes a bug in which we incorrectly determined paused dispatches, which led to not allowing multiple (overlapping) client pauses
2023-12-12 19:28:48 +03:00
Kostas Kyrimis
8640edad71
feat(acl): add acl keys to acl log command (#2274)
* add acl keys to acl log command
* add tests
2023-12-12 17:00:41 +02:00
Kostas Kyrimis
8323c82dc5
feat(acl): add acl keys to acl save/load (#2273)
* add acl keys to acl savel/load
* add tests
2023-12-08 16:08:33 +00:00
Kostas Kyrimis
2703d4635d
feat(acl): add validation for acl keys (#2272)
* add validation for acl keys
* add tests
2023-12-08 17:28:53 +02:00
Kostas Kyrimis
8126cf8252
feat(acl): add acl keys to acl list command (#2261)
* add acl keys to acl list
2023-12-08 15:32:15 +03:00
Vladislav
11ef6623dc
feat: DispatchTracker to replace everything (#2179)
* feat: DispatchTracker

Use a DispatchTracker to track ongoing dispatches for commands that change global state

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-12-05 11:02:11 +03:00
Borys
24b13434cf
feat: add slot-migration-status command (#2239)
* feat: add slot-migration-status command
2023-12-04 12:47:46 +02:00
Roman Gershman
26512fdba4
fix: remove string copy in SendMGetResponse (#2246)
fix: eliminate the redundant string copy in SendMGetResponse

Also, allow selectively create DflyInstance in pytests that is attached to
an existing dragonfly port, created outside of tests.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-12-03 18:14:19 +02:00
Borys
bfb1b3b624
Start slot migration (#2218)
* feat: add new command START-SLOT-MIGRATION
2023-11-29 13:38:13 +02:00
Roman Gershman
b853b2ab00
fix: memcached VERSION is now parseable by php-memcached client (#2220)
The DF version is being unparseable by Memcached::getVersion() that expects n.n.n string.
Change the version to emulate the old memcached server.
The DF version can still be fetched via Memcached::getStats() function.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-11-27 20:54:00 +02:00
Vladislav
d6044edbab
fix(squashing): Reset base command id (#2209)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-11-26 12:40:37 +02:00
Borys
e6f3522d59
fix: forbid parallel save operations (#2172)
* fix: forbid parallel save operations

* feat: add SAVE option to takeover command
2023-11-21 13:56:27 +02:00
Vladislav
604c600166
fix(pytest): Fix renamed flag (#2197)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-11-20 20:54:11 +00:00
Vladislav
d21f82a5f9
chore: connection fixes (#2192)
* chore: add more states to client connections

* fix: clear pipelined messages before close

* fix: skip same thread on backpressure
---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
Co-authored-by: Roman Gershman <roman@dragonflydb.io>
2023-11-20 17:08:12 +00:00
Kostas Kyrimis
4a1cb5bfa2
fix(memcached): add length check for key values (#2153)
* fix length checks for store commands
* add test
2023-11-20 14:37:29 +02:00
adiholden
c95f4961be
fix(server): client pause fix on pipeline squash (#2180)
* fix(server): client pause fix on pipeline squash

allow squashing commands on pause
move await on client pause inside InvokeCommand - this way all flows of command invoke will read pause state

Signed-off-by: adi_holden <adi@dragonflydb.io>
2023-11-16 13:30:02 +02:00
adiholden
b61d07d2c1
regression: skip client pause test utill we fix the bug (#2177)
Signed-off-by: adi_holden <adi@dragonflydb.io>
2023-11-15 16:31:45 +02:00
Roy Jacobson
c3a2da559e
feat(server): Implement CLIENT PAUSE (#1875)
* feat(server): Implement CLIENT PAUSE

Signed-off-by: adi_holden <adi@dragonflydb.io>
2023-11-15 08:56:49 +02:00
Kostas Kyrimis
09415c4f57
chore(tls): add tls config test for ca_dir (#2152)
This PR introduces a test case for TLS with `ca_dir`. First, we
did not have any tests for this case. Second, using `ca_dir` requires
to call `c_rehash` on the directory before it is loaded by DF. We
did not have this use case anywhere and therefore we thought there was
a bug when we used `ca_dir` only to find out that we need to call
`c_rehash` on the directory before we load the certificates. Now,
both a test and a use case are properly documented

* add missing test for ca_dir
* use rehash to properly show how to load ca directories instead of
  files
2023-11-13 14:11:14 +02:00
Vladislav
46292968ad
fix(search): Fix replication (#2159)
* fix(search): Support replication

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-11-13 11:58:54 +03:00
Vladislav
564e38c05c
chore: lower takeover test load, add comments (#2151)
* chore: lower takeover test load, add comments

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-11-12 12:08:05 +03:00
Kostas Kyrimis
5381746158
fix(regTests): increase cancel replication test timeout (#2143)
* increase timeout on cancel replication immediately 
* reduce the amount of commands run to 100 in the test
2023-11-08 23:00:00 +03:00
Vladislav
821884e333
chore(search): Extend FT.INFO (#2133)
* chore(search): Add index definition info to ft.info

* chore(search): Add flags to ft.info

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-11-06 16:18:13 +03:00
Kostas Kyrimis
2baadd1e90
fix(acl): case insensitive parsing from files and serialization format (#2123)
* replace > with # for acl files
* replace ACL SETUSER with USER for acl files
* add case insensitive parsing for acl files
* update tests
2023-11-05 11:43:11 +02:00
Roman Gershman
7aa3dba423
chore: use decode_responses when creating a redis client (#2109)
* chore: use decode_responses when creating a redis client

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-11-03 11:00:26 +02:00
Borys
2f39e89189
fix: add ability to set snapshot_cron flag during runtime (#2101)
* fix: add validating for snapshot_cron flag during runtime
* refactor: move warning log to upper level
2023-11-03 10:10:16 +02:00
Shahar Mike
169c9d3975
fix(regTests): Wait between ACTIVE until `stable_sync (#2111)
Regression test sometimes fails because for a short period of time after `wait_available_async()` returns, the result of `ROLE` could still be different from `stable_sync`

[Failure example](https://github.com/dragonflydb/dragonfly/actions/runs/6726461923/job/18282759612#step:6:1863)

We change our state from `LOADING` to `ACTIVE` [here](d08d7f13b4/src/server/replica.cc (L426)), but then we change the sync state 2 times:
1. `!R_SYNCING` [here](d08d7f13b4/src/server/replica.cc (L427C28-L427C37))
2. And only later to `R_SYNC_OK` (meaning `stable_sync`) [here](d08d7f13b4/src/server/replica.cc (L221))

This is easy to reproduce by adding a sleep right after the set of state to `ACTIVE`, either before or after the flipping of `R_SYNCING` (with different returned states).

BTW without that added sleep I was not able to reproduce, having tried 1000s of times in various configurations.

We could change the order of things such that we first change `state_mask_` and only then switch state from `LOADING` to `ACTIVE` (which is probably the right thing to do), but that would require a subtle refactor, as we change these in a couple of places.

But we should keep in mind that this has no effect on users. So a simple sleep on the test side should fix this fairly well.
2023-11-02 13:09:42 +02:00
Kostas Kyrimis
d08d7f13b4
fix(regTests): can't execute command while loading on snapshots (#2110) 2023-11-02 12:17:08 +02:00
Roman Gershman
8a65aec805
chore: help users to fix a mistake of setting quotes in the flagfile (#2092)
* chore: help users to fix a common mistake of setting quotes in the flagfile

Specifically, the confusion is often around the cron expression.
---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-10-30 22:59:00 +02:00
Kostas Kyrimis
84744a4004
fix(ReplyGuard): set member variable on construction and remove unused (#2082)
* set member variable to point to the reply builder
* remove redundant checks
* import missing signal in instance.py
2023-10-30 13:54:27 +02:00
Roman Gershman
967c6e0f6e
chore: improve pytest ux (#2088)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-10-30 09:01:23 +02:00
iko1
ac170748fb
feat(acl): add config set for acllog_max_len (#1979) 2023-10-29 18:17:02 +02:00
Vladislav
04cd2ff3f9
fix(search): Support indexing array paths (#2074)
* fix(search): Support indexing array paths

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>


---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-10-29 15:14:23 +03:00
Vladislav
474ea5137a
fix(replica): fix replica reconnect handing (#2068)
* fix(replica): fix replica reconnect handing

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-10-27 11:12:55 +03:00
Kostas Kyrimis
af622f6a52
fix(requirements): conflicting dependencies redis and redis-om (#2079) 2023-10-27 09:18:49 +03:00
Vladislav
5eed2bfe11
fix(search): Persist field flags (#2069)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-10-25 13:22:40 +03:00
Roman Gershman
dc025e5f94
fix: inject our own parser for slowlog get (#2059)
Also adjust debug logs to be less verbose.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-10-24 11:02:51 +03:00
Andy Dunstall
124bafc06b
feat(tls): support runtime tls reconfig (#2047)
* feat(tls): add tls reconfig

* feat(config): error if multiple config params given

* tls: move ctx ref to connection
2023-10-23 16:35:39 +00:00
Kostas Kyrimis
dd2cb1def1
fix(regTests): properly handle other error codes on timeouts (#2057) 2023-10-23 16:35:29 +03:00
Vladislav
67bb3970be
chore(search): Add exclusive range option (#2055)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-10-23 13:07:40 +03:00
Vladislav
2cb7d30603
fix: skip setting tcp_nodelay for unix domain sockets (#2033)
* fix handling of unix domain sockets

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-10-22 11:00:51 +03:00
Roman Gershman
83d5b849a3
fix: localize timeout in test_cancel_replication_immediately test (#2048) 2023-10-22 08:44:00 +03:00
Vladislav
9e8d886ce9
chore(search): add redis-om test (#2049)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-10-21 14:22:04 +03:00
Kostas Kyrimis
1d02e12ad1
chore: call debug stacktrace on SIGUSR1 (#2012)
* add macro to install a signal handler that prints the contents of debug stacktrace on SIGUSR1
* add this on regTests
2023-10-20 10:50:55 +03:00
Kostas Kyrimis
64841efeed
chore(regTests): print logs when regTests timeout (#2031)
* add a python script to print the most recent log
* if CI timeouts, print the most recent log
* replace global timeout with timeout command
* upload all logs on failure()
* print uid + port + the log files for each df instance
2023-10-20 10:50:19 +03:00
iko1
954e5a0a80
feat(acl): add config set for aclfile (#1978) 2023-10-19 10:15:38 +00:00
Uku Loskit
6a75c6ddc5
feat: Implement slowlog (#1956)
Implement slowlog

Signed-off-by: Uku Loskit <ukuloskit@gmail.com>
2023-10-13 23:39:35 +03:00
Kostas Kyrimis
76a030b1a7
chore(regTests): add basic module logging and print when tests fail (#2005)
* add vmodule level 1 logging for df instances
* print logs when df instance fails to stop
2023-10-12 14:38:35 +03:00
Shahar Mike
b1bd2103d7
opt(server): If unspecified, use num_shards == proactor_threads (#1998)
* opt(server): If unspecified, use num_shards == proactor_threads

* Use same config for tests, per Roman's request
2023-10-10 14:12:39 +03:00
Kostas Kyrimis
cbba6f4e11
fix(acl/admin_port): skip acl validation on admin port (#1997)
* skip all acl related functionality on admin port
2023-10-10 08:28:07 +03:00
Kostas Kyrimis
5471827448
update(requirepass): behave as redis requirepass (#1996)
* requirepass also updates ACL default user password
* update config set requirepass to include the new behaviour
* add tests
* fix non existent default user when loading empty files
2023-10-09 16:34:42 +03:00
Andy Dunstall
2d28b48481
feat(aws): add s3 awsv2 (#1929)
* feat(aws): add s3 awsv2

* feat(aws): add s3 snapshot test

* feat(aws): disable ec2 metadata by default

* feat(aws): add s3 disable payload signing flag

* chore: update helio

* fix: fix requirements.txt

* feat(s3): update sign payload flag

* chore: update helio
2023-10-06 10:24:56 +01:00
Kostas Kyrimis
0c1402c4ab
fix(regTests): test_cancel_replication_immediately races (#1990) 2023-10-06 09:38:14 +03:00
Kostas Kyrimis
e98d5e60c8
fix(regTests): adjust small network buff test (#1991)
* Seeder runs infinitely
2023-10-05 16:37:40 +03:00
Kostas Kyrimis
8b7a43d214
fix(Listener): skip auth when requirepass is empty on http (#1985)
* Fix a small bug on http when username was required even if requirepass was empty
2023-10-03 19:56:19 +03:00
Vladislav
1ea0c39d6a
fix(server): Fix async lua (#1982)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-10-03 16:13:38 +03:00
Vladislav
20b924f9d5
fix(regtests): Reduce load on redis replication test (#1984)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-10-03 12:28:42 +03:00
Kostas Kyrimis
945d3a39e4
chore(Http): change authed username to default and skip auth on metrics (#1950)
* Update Http auth with username default instead of user
* skip auth for /metrics page
* add/improve tests
* fix a bug with admin port requiring auth on http even if nopass was set
* update helio ref
* update listener class to contain its respective Role
* fix http init to only include admin and main listener
2023-10-03 10:45:37 +03:00
Kostas Kyrimis
59dd48d47a
fix(RegTests): fix replication test_network_disconnect_active_stream (#1980)
* The issue was similar with test `network_disconnect_small_buffers` but this time the debug build could be slow enough for the replication to not finish. As a consequence, by the time the test reached the assertion, the log did not contain the expected output.
2023-10-02 17:35:25 +03:00
adiholden
84d4ba4d69
chore(regression): test bptree on regression pytests (#1963)
* chore(regression): test bptree on regression pytests

1. stop passing the flag use_zset_tree as it is true on default
2. fix ci test to run replication tests
3. change replication tests seeder to sometimes add more than 128 values
   to zset to test the pbtree impl

Signed-off-by: adi_holden <adi@dragonflydb.io>
2023-10-02 14:07:50 +00:00
Roman Gershman
57441f9863
chore: Print useful info when printing DflyInstance objects (#1977) 2023-10-02 12:02:46 +03:00
Vladislav
bc48bed6ad
fix(regtests): Colored per-instance log (#1971)
* fix(regtests): Colored per-instance log

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-10-02 10:36:41 +03:00
Andy Dunstall
e6b8cd1d76
feat(cmd): add restricted commands flag (#1967) 2023-09-29 16:16:06 +01:00
Kostas Kyrimis
ba4eba991d
fix(regTests): adjust network disconnect small buffer to pass on (#1970) 2023-09-29 16:50:01 +03:00
Roy Jacobson
7ad29ab919
fix(regtests): Limit infinite loop in replication_test (#1961) 2023-09-28 11:45:03 +03:00
Roy Jacobson
5c9c9255d2
chore: Small refactor of DflyInstance (#1951)
* Move to its own file
* Unify self.args and self.params.args earlier so it can be inspected.
2023-09-28 10:11:11 +03:00
Kostas Kyrimis
949cedf66c
fix(AclFamily): do not allow to delete default user (#1954)
* do not allow to delete default user
* upon loading an acl file, if default does not exist create them
* add test
2023-09-28 09:22:45 +03:00
Kostas Kyrimis
b82825cd6c
fix: test_cancel_replication_immediately (#1942)
* Give enough time to get replication canceled when the CI server is under heavy load
2023-09-27 17:56:26 +03:00
Roy Jacobson
ab903612f1
fix(regtests): Bind to 127.0.0.1 and not localhost (#1937)
* fix(regtests): Bind to 127.0.0.1 and not localhost

* Split start into start/serve
2023-09-27 14:58:52 +03:00
Roy Jacobson
d50b492e1f
feat(replication): First iteration on partial sync. (#1836)
First iteration on partial sync.
2023-09-26 10:35:50 +03:00
Vladislav
fc0943989e
feat(search): return scores (#1870)
* feat(search): return scores

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-09-25 10:03:17 +03:00
Vladislav
bb77de7551
fix(regests): Fix port detection with gdb (#1922)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-09-24 15:42:31 +03:00
Vladislav
9ccdfb4bf1
fix(regtests): filter client setinfo from rewrite log (#1921)
Fix tests failing after client update to 5.0
2023-09-24 10:51:56 +03:00
Vladislav
d8b99dce93
chore(regtest): Update redis dependency (#1915)
* chore(regtest): Update redis dependency

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-09-23 10:06:21 +03:00
Vladislav
5bc069f37e
fix: Correctly persist vector indices (#1905)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-09-22 11:29:19 +03:00
Kostas Kyrimis
cfd83a66a2
fix(regression-tests): increase waiting delay for df startup (#1893) 2023-09-19 19:42:12 +03:00
Kostas Kyrimis
890761989c
feat(AclFamily): add acl log (#1865) 2023-09-18 20:10:53 +03:00
Roy Jacobson
1e61ec8114
chore: Add a context manager to DflyInstance so we don't forget to close them. (#1873)
* chore: Add a context manager to DflyInstance so we don't forget to close
them.

* Update tests/dragonfly/config_test.py

Co-authored-by: Roman Gershman <roman@dragonflydb.io>
Signed-off-by: Roy Jacobson <roi.jacobson1@gmail.com>

---------

Signed-off-by: Roy Jacobson <roi.jacobson1@gmail.com>
Co-authored-by: Roman Gershman <roman@dragonflydb.io>
2023-09-18 13:52:56 +03:00
Roy Jacobson
74d78261fc
chore: Fix regression test after #1569 (#1872) 2023-09-18 12:26:18 +03:00
Roy Jacobson
b55316c0e6
tests: choose open ports randomally (#1569)
* Implement changes to the testing infrastructure to use random ports
* Use psutil to find out the random ports
2023-09-18 10:23:49 +03:00
Uku Loskit
7a5fe1adc1
fix: fix json.arrappend not allowing passing JSON objects (#1867)
Signed-off-by: Uku Loskit <ukuloskit@gmail.com>
2023-09-16 12:27:20 +03:00
Kostas Kyrimis
bbd4c6b636
feat(AclFamily): add acl commands (#1844) 2023-09-15 14:28:36 +03:00
Kostas Kyrimis
ff079f0af1
fix(ServerFamily): proper initialization of FileSnapshotStorage when replicaof flag is used (#1858)
* fix(ServerFamily): proper initialization of FileSnapshotStorage when
replicaof flag is passed
2023-09-15 09:00:56 +03:00
Andy Dunstall
a8f3764048
feat(tests): unset test DFLY env variables (#1856) 2023-09-13 18:24:48 +01:00
Aidarbek Suleimenov
04e88a1381
feat(server): Getting flags from the environment variables (#1810)
* flags from env variables

* querying environment vars

* remove includes

* refactor

* exit for unknown flag with DFLY_ prefix

* reflecting change in the test

* better tests

* refactor + new test case

* refactor test with inner class

* refactor

* revert back test flags as it might affect ci/cd

* fixing flags

* refactor

* remove includes

* refactor
2023-09-13 10:02:38 +00:00
Roman Gershman
99244b1049
Support 'databases' alias for CONFIG GET (#1849)
Fixes #1842

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-09-12 19:19:05 +03:00
Kostas Kyrimis
684bf97ce6
fix(AclFamily): move acl test files to pytest (#1843)
* Fixes broken release build
* Moves acl test files to acl pytest
2023-09-12 09:38:58 +03:00
adiholden
ba8adf7bde
skip test_multidim_knn (#1834)
Signed-off-by: adi_holden <adi@dragonflydb.io>
2023-09-10 08:35:42 +00:00
Kostas Kyrimis
fc6b7e5f9a
fix(regression-tests): correct path for acl files (#1829) 2023-09-09 11:59:17 +03:00
Kostas Kyrimis
164b045505
fix: regression tests for acl family test (#1826) 2023-09-08 19:57:09 +03:00
Kostas Kyrimis
48488c5682
feat(AclFamily): load/store aclfile (#1820)
* add ACL LOAD
* add ACL SAVE
* add --aclfile command
2023-09-08 14:20:06 +03:00
Vladislav
e69f18286b
feat(search): HNSW (#1799)
* feat(search): HNSW

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-09-08 09:57:18 +03:00
Kostas Kyrimis
d1002842b8
chore(AclFamily): Move some of the regressions tests to unit and add (#1816)
* move error handling tests from regression to unit
* move ACL LIST regression to unit test
* move AUTH regression to unit test
* move ACL WHOAMI regression to unit test
* add unit tests for SETUSER/DELUSER (so they run on every PR)
* add unit tests for all ACL categories
2023-09-06 16:03:22 +00:00
Roy Jacobson
502f76fada
Enable authentication for the HTTP interface. (#1792) 2023-09-04 15:18:57 +03:00
Vladislav
f1d9ab30ee
feat(search): Query parameters (#1768)
* feat(search): Query parameters

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-09-03 23:08:25 +03:00
Kostas Kyrimis
196650a39d
chore(AclFamily): disable flaky tests (#1791) 2023-09-03 09:23:14 +00:00
Kostas Kyrimis
846d421d25
fix(AclFamily): setuser interleaved signed categories update (#1790) 2023-09-03 09:02:32 +00:00
Kostas Kyrimis
68fa3f420b
feat(AclFamily): add acl whoami command (#1774)
* add acl whoami
* add tests
2023-09-01 21:23:01 +03:00
Kostas Kyrimis
6706707585
feat(AclFamily): add acl deluser (#1773)
* add acl deluser command
* add tests
2023-09-01 16:12:02 +00:00
Kostas Kyrimis
9ca7dba143
fix(AclFamily): stream acl updates via dispatch queue in connection (#1786)
There was a bug on updates of the acl categories when squashing was used. Basically, the parent context could be accessed in parallel by the "stub" contexts causing a dreaded data race on the update.

This is fixed by adding a new AclUpdateMessage at the front of the dispatch queue of the connection.
2023-09-01 15:40:50 +00:00
Vladislav
eaedc96dc5
fix(search): return whole json documents (#1782)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-09-01 12:50:11 +03:00
Kostas Kyrimis
4350cf35d7
fix: failing memcached regression test (#1778) 2023-08-31 12:59:50 +03:00
Roman Gershman
3829e3691e
chore: move rdb save related logic into auxillary file (#1769)
1. No logic was changed during refactoring.
2. Flipped the flag to run regression tests for now own with zset_tree=on

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-08-30 16:41:17 +03:00
Kostas Kyrimis
7c43cbf2b5
feat: add validation of acl users (#1743)
* add validation for categories
* add tests
2023-08-29 18:52:46 +03:00
Kostas Kyrimis
1855c1cd54
fix: broken memcached error reporting (#1741)
* fix DispatchCommand error reporting when memcached protocol is used (one example is when we use SET command on the replica -- previously we crashed now we properly report an error)
* SendError(ErrorReply) moved to SinkReplyBuilder from RedisReplyBuilder
* SendError(OpStatus) moved to SinkReplyBuilder from RedisReplyBuilder
* added tests for SendError(ErrorReply) in RedisReplyBuilder
2023-08-28 17:28:42 +00:00
Roman Gershman
af434cf710
feat: implement CONFIG GET command (#1751)
* feat: implement CONFIG GET command

The command returns all the matched arguments and their current values.
In addition, this PR adds mutability semantics to each config - whether it can be
changed at runtime.

Fixes #1700

Signed-off-by: Roman Gershman <roman@dragonflydb.io>

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-08-28 19:26:38 +03:00
Roy Jacobson
ed845fe526
feat(server): Support limiting the number of open connections. (#1670)
* feat(server): Support limiting the number of open connections.

* * Update helio after the small fix was merged to master
* Don't limit admin connections (and add a test case)

* Resolve CR comments
2023-08-27 11:30:16 +03:00
Roman Gershman
6dd51de9fe
fix: fix memcache bugs (#1745)
1. If the first request sent to the connection is large (2kb or more)
   Dragonfly was closing the connection.
2. Changed server side error reporting according to memcache protocol:
   https://github.com/memcached/memcached/blob/master/doc/protocol.txt#L172
3. Fixed the wrong casting in DispatchCommand.
4. Remove practically unused code that translated opstatus to strings.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-08-27 11:29:01 +03:00
Kostas Kyrimis
50545fc176
feat(AclFamily): add AUTH for acl members (#1732)
* Extends AUTH command to authenticate ACL users
* Add tests
2023-08-25 18:56:15 +03:00
Vladislav
3520697b22
fix: fix search test (#1739)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-08-25 10:01:36 +03:00
Vladislav
84871b8dce
feat: search index persistence (#1721)
* feat: WIP search index persistence

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>

* Update src/server/search/doc_index.cc

Co-authored-by: Kostas Kyrimis  <kostaskyrim@gmail.com>
Signed-off-by: Vladislav <vladislav.oleshko@gmail.com>

* fix: foxes

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
Signed-off-by: Vladislav <vladislav.oleshko@gmail.com>
Co-authored-by: Kostas Kyrimis <kostaskyrim@gmail.com>
2023-08-24 13:41:10 +03:00
Kostas Kyrimis
bd87fb75fa
feat(AclFamily): add acl setuser command (#1725)
* Add `ACL SETUSER` command
* Add tests
2023-08-24 13:24:18 +03:00
Kostas Kyrimis
7f89bf37d8
fix(AclFamily): commands parsing and error handling (#1726)
* Fixed broken ACL command (prev df crashed)
* Fixed broken ACL LIST STR (now prints error)
* Added tests
2023-08-23 14:23:41 +03:00
Roy Jacobson
331e6a4d47
chore: Mark slow pytests as 'slow' so they can be excluded. (#1720) 2023-08-23 13:04:49 +03:00
Kostas Kyrimis
898061d738
feat(AclFamilly): add acl list command (#1722)
* Add acl-family source and header
* Add `ACL LIST` command
* Add a simple test to check the default user
2023-08-22 15:33:14 +00:00
Vladislav
ac79167530
fix: Add small timeout to monitor (#1718)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-08-21 10:20:43 +00:00
Vladislav
e0f3684e5d
feat: json paths in search (#1695)
* feat: json paths in search

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
Signed-off-by: Vladislav <vladislav.oleshko@gmail.com>
Co-authored-by: Roy Jacobson <roi.jacobson1@gmail.com>
2023-08-18 15:40:59 +03:00
Vladislav
c65b9cf63d
fix: Fix squashing, pytest arg formatting (#1712)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-08-18 09:28:19 +03:00
Vladislav
4fbd0e38dd
feat: Pipeline squashing (#1619)
* feat: Pipeline squashing

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
Signed-off-by: Vladislav <vladislav.oleshko@gmail.com>
Co-authored-by: Kostas Kyrimis <kostaskyrim@gmail.com>
2023-08-17 16:06:48 +03:00
Vladislav
71fa2f275e
fix: MONITOR now works for multi transactions (#1675)
* fix: fix monitoring for multi transactions

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-08-17 12:50:16 +03:00
talbii
16c2353faf
introduce --replicaof flag (#1583)
* introduce `--replicaof` flag

Closes #1381.

The behvaiour of `--replicaof` is similar to `REPLICAOF`. On startup, the instance continuously attempts to connect to master. Stop using the normal `REPLICAOF NO ONE` command.

The flag expects format `<IPv4/host>:<port>` or `[<IPv6>]:<port>`.

---------

Signed-off-by: talbii <ido@dragonflydb.io>
Signed-off-by: talbii <41526934+talbii@users.noreply.github.com>
2023-08-09 14:42:08 +03:00
Vladislav
7f1a2d9e38
fix(pytest): Raise exception if instance.stop timed out (#1660)
fix(pytest): Raise if instance.stop timed out

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-08-07 19:10:15 +03:00
Vladislav
279d7cc068
feat: FtList, FtDropIndex and FtInfo (#1649)
* feat: FtList, FtDropIndex and FtInfo

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-08-06 22:01:58 +03:00
Dor Avrahami
0b6c0bbe53
feat: support for cron expressions based snapshot (update). (#1620)
Requested by #1590.
Introducing a new flag --snapshot_cron, enabling users to use cronjob expressions to time snapshot saves.
Cronjob expressions are parsed using a third party library croncpp.
This PR continues #1599, updating cron expressions to crontab style,
up to minutes resolution instead of seconds.

Signed-off-by: Dor Avrahami <da19965@gmail.com>
2023-08-02 12:58:45 +03:00
Shahar Mike
3b0bd212f4
fix(test): Fix failing tests. (#1612)
Solution is to wait until snapshot is ready, instead of hard coding a
sleep schedule. Also don't reuse files by other test cases.
2023-08-01 08:30:17 +00:00
Dor Avrahami
977fc18e25
feat: support for cron expressions based snapshot. (#1599)
Introducing a new flag `--snapshot_cron`, which enables users
to use cron expressions to time snapshot saves.

Signed-off-by: Dor Avrahami <da19965@gmail.com>
2023-07-31 20:26:01 +03:00
Roy Jacobson
4c85d5825d
tests: Add a password to TLS configurations (#1603)
Add a password to TLS configurations
2023-07-31 08:48:36 +00:00
adiholden
3ed0bb480f
Fix(regression test): fix test_flushall_in_full_sync (#1597)
* Fix(regression test): fix test_flushall_in_full_sync

The bug: the test checks the replication using role command on replica
The replica updates the status to full sync when starting the full sync
flow, but actually the master did not start snapshoting yet.
The fix: check the status using role command on master, because master
updates the status only after snapshoting started.

Signed-off-by: adi_holden <adi@dragonflydb.io>
2023-07-31 09:24:37 +03:00
Roy Jacobson
85619e0010
sec: Adjust flag validation for TLS. (#1582)
* sec: Adjust flag checks when using TLS.

* Trust default certificates if no specific roots are given

* Add regression tests for the different scenarios

* Validate that client connections work as well
2023-07-30 19:24:39 +02:00
adiholden
57dc42530e
Fix(regression test): test_tls_replication (#1580)
The test fails sometimes when starting master after killing it.
The reason for this is that OS did not release port untill we started
master again.
The fix - adding sleep after kill
After we will have randomly selected ports on pytest we can remove this
sleep.

Signed-off-by: adi_holden <adi@dragonflydb.io>
2023-07-23 10:27:12 +03:00
Kostas Kyrimis
9698d6fea2
test(replication): add admin port replication tests (#1561) 2023-07-20 14:35:41 +03:00
Kostas Kyrimis
078d152ae0
feat: add replication over tls (#1525)
1. Introduces `tls_replication` flag to allow tls connections for replicas
2. Add pytests
2023-07-19 18:21:46 +00:00
Kostas Kyrimis
7944af3c62
feat: Add black formatter to the project (#1544)
Add black formatter and run it on pytests
2023-07-17 13:13:12 +03:00
adiholden
c27fa8d674
fix(regression test): fix in shutdown and replication pytests (#1530)
* fix(regression_test): fix in shutdown and replication pytests

- skip test_gracefull_shutdown test
- fix test_take_over_seeder test:
    bug: the dbfilename was not unique, therefore between different runs the server reload
    the snapshot of the last test run and this failed the test.
    fix: use random dbfilename
- fix test_take_over_timeout test:
    bug: REPLTAKEOVER timeout was not small enough for opt dfly build
    fix: decrease timeout

Signed-off-by: adi_holden <adi@dragonflydb.io>
2023-07-11 09:56:20 +03:00
Kostas Kyrimis
77a223d36d
fix: add tls-ca-cert-file and tls-ca-cert-dir flags to allow tls certificate validation (#1515)
1. add tls-ca-cert-file flag
2. add tls-ca-cert-dir flag
3. enables redis-cli to connect over tls without --insecure flag by properly validating certificate wtih CA
2023-07-11 08:28:18 +03:00
Shahar Mike
a6745850ab
Fix regression test failures. (#1529)
The issue was that, sometimes, the ID generated for one of the nodes
contained the slot ID that was used in the test (either 5259 or 5260).
This caused the test to replace the "slot" part of the id, which in turn
caused the node to think that it no longer owns any slot.
2023-07-10 10:59:58 +03:00
Roy Jacobson
3904a4f628
Fix regression test failures on old Python versions (#1521)
Signed-off-by: adiholden <adi@dragonflydb.io>
Co-authored-by: adiholden <adi@dragonflydb.io>
2023-07-06 15:27:39 +03:00
Kostas Kyrimis
15481b81ce
feat(replication): allow non-tls connections between replica and master on admin port #1419 (#1490)
1. Add new flag no_tls_on_admin_port
2. Add replication tests for no_tls_on_admin_port
2023-07-06 14:04:45 +03:00
Roy Jacobson
0f69d32b11
takeover: Cancel blocking commands (#1514)
* fix: Cancel blocking commands when performing a takeover

* Add some comments

* Make CancelBlocking a method of ConnectionContext

* add a small todo
2023-07-05 17:09:10 +02:00
Roy Jacobson
177a21b266
Fix a bug and add a timeout test for takeover. (#1512) 2023-07-03 16:46:38 +02:00
Roy Jacobson
4babed54d3
feat: Support atomic replica takeover (#1314)
* fix(server): Initialize ServerFamily with all listeners.

- Add a test for CLIENT LIST which is the visible result of this.

* use std move

* feat: Implement replicas take over

* Basic test

* Address CR comments

* Write a better test. Sadly it fails

* chore: Expose AwaitDispatches for reuse in takeover

* Ensure that no commands can execute during or after a takeover

* CR progress

* Actually disable the expiration

* Improve tests coverage

* Fix the dispatch waiting code

* Improve testing coverage and fix a shutdown snaphot bug

* don't replicate a replica
2023-07-02 16:11:28 +02:00
Roy Jacobson
52192e0596
fix(server): Initialize ServerFamily with all listeners. (#1485)
* fix(server): Initialize ServerFamily with all listeners.

- Add a test for CLIENT LIST which is the visible result of this.

* use std move
2023-07-02 10:01:54 +02:00
Vladislav
1ee0e30255
fix: disable prohibited eval in multi test (#1505)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-07-02 10:50:03 +03:00
Vladislav
cfca751848
feat: global eval in exec (#1443)
Enables execution of global lua scripts inside multi/exec transactions if the defualt script config enables global execution for scripts. This change is only a fix and does not provide any safeguards against other execution scenarios (namely enabling globality with script flags). In the future, the proper execution mode should be determined more carefully by inspecting the scripts to be executed

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
Co-authored-by: Kostas Kyrimis  <kostaskyrim@gmail.com>
2023-07-01 22:12:05 +03:00
Kostas Kyrimis
542b9783b7
fix(regression): remove test case for info persistence (#1492)
The test case for checking is_loading == 1 is inherently racy because
the client can connect at any time before or after the dragonfly
instance loads the snapshot.
2023-07-01 11:57:54 +03:00
Kostas Kyrimis
3eaeef096c
fix: misspelled replication on pytest redis_replication (#1501) 2023-07-01 11:56:43 +03:00
Shahar Mike
5c11beb919
feat(cluster): Send flush slots cmd from masters to replicas. (#1484)
This fixes potential data diffs between master and replica upon slot moving

Fixes #1320
2023-06-28 22:04:51 +03:00
Roy Jacobson
cbe72e353d
fix: Increase the duration of the seeding in test_replication_info. (#1479) 2023-06-26 10:41:25 +03:00
Kostas Kyrimis
0670a81488
fix(regression): failing info persistence on snapshot pytests (#1473) 2023-06-23 16:04:44 +03:00
Kostas Kyrimis
10479a4fba
fix: failing assertion on snapshot regression tests when dbfly instance shuts down (#1463)
This PR is a temporary solution for clients that are not properly
removed from the connection pool triggering an active client assertion
during dragonfly instance shutdown
2023-06-22 13:47:36 +03:00
Roy Jacobson
0a7328f1e1
Expose replication lag metrics (#1400)
* feat(server): Expose lag metrics over prometheus.

---------

Signed-off-by: Roy Jacobson <roy@dragonflydb.io>
2023-06-22 11:12:41 +02:00
Vladislav
fea4e1dd4d
fix: rename memcache to memcacheD (#1461)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-06-22 11:55:20 +03:00
adiholden
98e84f804a
feat(regression github): send failed tests names to chat (#1459)
* feat(regression) : send failed tests names

Signed-off-by: adi_holden <adi@dragonflydb.io>
2023-06-22 09:47:12 +03:00
Kostas Kyrimis
99f3284910
feat(server): Add missing fields to INFO PERSISTENCE command (#1408) (#1438)
* Add loading field
* Add rdb_changes_since_last_save field
2023-06-21 23:35:36 +03:00
Vladislav
f25098bb98
fix: add numpy to regtests (neede for knn) (#1454)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-06-21 09:18:42 +03:00
Vladislav
a9d9b4935c
feat: Implement KNN interface (#1412)
Add external KNN interface

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-06-20 14:10:08 +03:00
Vladislav
6d4d740d6e
fix: Don't remove non-concluding tx from queue on ooo runs (#1427)
* fix: Don't remove non-concluding tx from queue on ooo runs

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-06-18 21:14:28 +03:00
Roman Gershman
69e6ad799a
fix: remove bad check-fail in the transaction code (#1420)
fix: remove bad check-fail in the transaction code.

Fixes #1421.

The failure reproduces for dragongly running with a single thread where all the
arguments grouped within the same ShardData

Also, we improve verbosity levels inside reply_builder.cc.
For that we extend SinkReplyBuilder to support protocol errors reporting
and we remove ad-hoc code for this from dragonfly_connection.
Required to track errors easily with `--vmodule=reply_builder=1`

Finally, a pytest is added to cover the issue.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2023-06-18 07:03:08 +03:00
Roy Jacobson
f8d4b6f7c4
Put more keys, make it non-failure if the test fails because of bad params (#1411) 2023-06-14 22:00:12 +03:00
Roy Jacobson
46bf0bb42c
fix(test): Fix a race condition in test_flushall_in_full_sync (#1409) 2023-06-14 13:00:46 +02:00
Vladislav
2a5fd79856
fix: Allow readonly commands in replica script (#1392)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-06-12 06:14:17 +03:00
Roy Jacobson
9c8b3296b2
fix: A more gracefull shutdown (#1388)
* Add a test for gracefull shutdown

* Add a small wait loop for dispatching connections
2023-06-11 23:22:27 +03:00
Vladislav
e837b3d229
Fix reply builder access issue (#1378)
* fix: Fix invalid reply builder use

---------

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-06-10 00:50:05 +03:00
Chaka
de21afc6e6
test(cluster): Test that cluster works with a standard cluster client. (#1336)
In this case, `redis.RedisCluster`.

To be double sure I also looked at the actual packets and saw that the
client asks for `CLUSTER SLOTS`, and then after the redistribution of
slots, following a few `MOVED` replied, it asks for the new slots again.
2023-06-06 16:03:09 +03:00
Chaka
5234f77727
fix(replication): Restart replication upon receiving FLUSHALL during full sync (#1347)
Fixes #1231
2023-06-06 12:26:47 +03:00
Vladislav
bf44b56667
fix: fix search_test after incorrect rebase (#1356) 2023-06-05 15:38:40 +03:00
Vladislav
9ab70e4f15
feat: Support tags in search (#1341)
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2023-06-05 00:26:21 +03:00