1
0
Fork 0
mirror of https://github.com/dragonflydb/dragonfly.git synced 2024-12-14 11:58:02 +00:00
Commit graph

2936 commits

Author SHA1 Message Date
Roman Gershman
5fee668391
chore: add active time to stream consumers (#4285)
* chore: add active time to stream consumers

Adjust XINFO command to output active-time property.
Store active-time and switch to RDB_TYPE_STREAM_LISTPACKS_3 if FLAGS_stream_rdb_encode_v2
is enabled.
---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-12-10 14:11:20 +02:00
Shahar Mike
0562796cac
fix: Do not attempt to defrag StringSet as a StringMap (#4283)
That'd be a total waste of time and energy, not to mention you'll crash.

Fixes #4167
2024-12-10 08:44:52 +00:00
Roman Gershman
44e27efc00
chore: fixes the parse error for xread/xreadgroup with unbalanced ids (#4266)
Partially addresses #4193

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-12-10 10:25:19 +02:00
Roman Gershman
f428dc31be
fix: support loading of 7.x streams correctly (#4281)
Now rdb_load supports RDB_TYPE_STREAM_LISTPACKS, RDB_TYPE_STREAM_LISTPACKS_2 and RDB_TYPE_STREAM_LISTPACKS_3 formats.
rdb_save still saves with RDB_TYPE_STREAM_LISTPACKS format - we want to release the DF version that can load everything first, and
then update the replicaion format in the next versions.

Also, update rdb_test.cc

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-12-10 09:57:36 +02:00
Roman Gershman
4428480a4e
chore: update command interface for main_service commands (#4265)
Clean up command_registry interface as well.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-12-09 23:09:22 +00:00
Roman Gershman
bf4f45a7c2
chore: let resp parser provide more useful logs (#4273)
* chore: let resp parser provide more useful logs

1. More warning logs around bad BAD_ARRAYLEN messages
2. Lift the restriction around big bulk strings and log a warning instead.
3. Pull helio

Probably fixes #4213

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-12-09 18:12:07 +02:00
adiholden
03d679ac31
fix(server) : dont apply eviction on rss over limit (#4276)
Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-12-09 14:19:25 +02:00
adiholden
d2f479b5da
fix cluster: migration traverse bug (#4279)
Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-12-09 13:22:02 +02:00
Borys
51e16b2ceb
fix: prohibit read commands during takeover (#4267) 2024-12-09 13:02:15 +02:00
adiholden
330d007d56
feat server: support config set serialization_max_chunk_size (#4274)
Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-12-09 09:12:18 +02:00
Shahar Mike
bafd8b3b8b
chore: Fix test_rss_used_mem_gap for all types (#4254)
* chore: Fix `test_rss_used_mem_gap` for all types

The test fails when it checks the gap between `used_memory` and
`object_used_memory`, by assuming that all used memory is consumed by
the `type` it `DEBUG POPULATE`s with.

This assumption is wrong, because there are other overheads, for example
the dash table and string keys.

The test failed for types `STRING` and `LIST` because they used a larger
number of keys as part of the test parameters, which added a larger
overhead.

I fixed the parameters such that all types use the same number of keys,
and also the same number of elements, modifying only the element sizes
(except for `STRING` which doesn't have sub-elements) so that the
overall `min_rss` requirement of 3.5gb still passes.

Fixes #3723

* threshold

* list

* comments test assert

* previous numbers

* ???
2024-12-08 13:34:59 +02:00
Borys
32ad00b09a
fix: close socket to prevent onCompletion call after the journal stre… (#4270)
fix: close socket to prevent onCompletion call after the journal streamer is destroyed
2024-12-08 11:57:08 +02:00
Kostas Kyrimis
f9f93b108c
chore: monotonically increasing ports for cluster tests (#4268)
We have cascading failures in cluster tests because on assertion failures the nodes are not properly cleaned up and subsequent test cases that use the same ports fail. I added a monotonically increasing port generator to mitigate this effect.
2024-12-06 12:07:23 +02:00
Andy Dunstall
63ccbbc0a7
fix(geo): support case insensitive units (#4264) 2024-12-05 15:01:23 +00:00
Borys
17651b2610
fix: test_network_disconnect_during_migration data size was too big f… (#4260)
fix: test_network_disconnect_during_migration data size was too big for timeout
2024-12-05 13:04:58 +00:00
Stepan Bagritsevich
5483d1d05e
fix(eviction): Tune eviction threshold in cache mode (#4142)
* fix(eviction): Tune eviction threshold in cache mode

fixes #4139

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor: small fix in tiered_storage_test

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor: address comments

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* chore(dragonfly_test): Remove ResetService

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor: fix test_cache_eviction_with_rss_deny_oom test

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor: address comments

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* fix(dragonfly_test): Fix DflyEngineTest.Bug207

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* fix(dragonfly_test): Increase string size in the test Bug207

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor: address comments 3

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor: address comments 4

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* fix: Fix failing tests

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor: address comments 5

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor: resolve conficts

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

---------

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2024-12-05 12:26:59 +00:00
Kostas Kyrimis
267d5ab370
chore: remove DbSlice mutex and add ConditionFlag in SliceSnapshot (#4073)
* remove DbSlice mutex
* add ConditionFlag in SliceSnapshot
* disable compression when big value serialization is on
* add metrics

---------

Signed-off-by: kostas <kostas@dragonflydb.io>
2024-12-05 13:24:23 +02:00
Kostas Kyrimis
7ccad66fb1
feat: add support for big values in SeederV2 (#4222)
* add support for big values in SeederV2

---------

Signed-off-by: kostas <kostas@dragonflydb.io>
2024-12-05 08:47:41 +00:00
Andy Dunstall
ad73e18f6d
fix(save): don't create dirs for cloud dir (#4259) 2024-12-05 08:39:47 +00:00
Borys
071e299971
refactor: remove redundant allocations for streamer (#4225)
* refactor: remove redundant allocations for streamer
2024-12-05 08:15:31 +00:00
Roman Gershman
c2f8349c51
chore: update command interface for cluster and search families (#4258) 2024-12-05 09:12:50 +02:00
Stepan Bagritsevich
81079df0e1
fix(rax_tree): Fix crash caused by destructor in RaxTreeMap (#4228)
* fix(rax_tree): Fix double raxStop call in the SeekIterator

fixes dragonflydb#4172

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

* refactor(rax_tree): Address comments

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>

---------

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2024-12-04 21:34:15 +04:00
Kostas Kyrimis
d8fda40d4d
chore: split RecordExpiry preemptive and non-preemptive flows (#4252)
* add FiberGuard to RecordExpiry for non-preemptive flows

---------

Signed-off-by: kostas <kostas@dragonflydb.io>
2024-12-04 15:46:00 +02:00
Roman Gershman
892a415201
chore: update command interface for acl, hll families (#4253) 2024-12-04 13:15:09 +00:00
adiholden
7a23ec2aac
fix(server): fix memory leak on lua error (#4236)
The bug:
calling lua_error does not return, instead it unwinds the Lua call stack until an error handler is found or the
script exits. This lead to memory leak on object that should release memory in destructor.
Specific example is the absl::FixedArray<string_view, 4> args(argc); which allocates on heap if argc > 4. The free was not called leading to memory leak.
The fix:
Add scoping to to the function so that the destructor is called before calling raise error

Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-12-03 16:47:43 +02:00
Roman Gershman
1c09056ab4
fix: command interface for json_family (#4243) 2024-12-03 16:13:14 +02:00
Roman Gershman
8d343bfd69
fix: stream bugs (#4240)
This PR syncs some of the improvements that were introduced in streams in Redis 7.2.3 OSS.

1. verify xsetid against max deleted id in the stream
2. Implement precise memory measurement of streams for "memory usage" command.
3. Always compact nodes in stream listpacks after creating new nodes.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-12-03 16:12:54 +02:00
Shahar Mike
95f2320825
chore: Hide managed service info in INFO (#4248)
Specifically:
* `INFO REPLICATION` does not list the replicas, but does still show
  `connected_slaves`
* `INFO SERVER` does not show `thread_count` and `os`

Fixes #4173
2024-12-03 16:09:13 +02:00
Shahar Mike
b0d633fb61
chore: Hide replica info in real cluster if --managed_service_info (#4241)
So far we only handled emulated cluster. This PR adds real cluster
support.

Related to #4173
2024-12-02 20:22:07 +02:00
Borys
935ae86c94
fix: RdbTest.LoadStream3 incorrect file usage (#4242)
fix: RdbTest.LoadStream3 used incorrect file
2024-12-02 17:58:18 +02:00
Shahar Mike
779bba71f9
fix: Fix test_network_disconnect_during_migration test (#4224)
There are actually a few failures fixed in this PR, only one of which is a test bug:

* `db_slice_->Traverse()` can yield, causing `fiber_cancelled_`'s value to change
* When a migration is cancelled, it may never finish `WaitForInflightToComplete()` because it has `in_flight_bytes_` that will never reach destination due to the cancellation
* `IterateMap()` with numeric key/values overrode the key's buffer with the value's buffer

Fixes #4207
2024-12-02 15:55:23 +02:00
Roman Gershman
dcee9a9874
fix: bugs in stream code (#4239)
* fix: bugs in stream code

1. Memory leak in streamGetEdgeID
2. Addresses CVE-2022-31144
3. Fixes XAUTOCLAIM bugs and adds tests.
4. Limits the count argument in XAUTOCLAIM command to 2^18 (CVE-2022-35951)

Also fixes #3830

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Signed-off-by: Roman Gershman <romange@gmail.com>
Co-authored-by: Shahar Mike <chakaz@users.noreply.github.com>
2024-12-02 15:12:14 +02:00
Roman Gershman
91aff49fcd
chore: fix bugs in stream_family (#4237)
1. Use transaction time in streams code, similarly to how we do it in other commands.
   Stop using mstime() and delete unused redis code.
2. Check for sequence overflow issue when passing huge sequence ids.
   Add a test.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-12-02 11:57:31 +02:00
andreibe
ada96d9041
Feat(Helm chart): Add env vars into pods (#4196)
* feat: update pod template

Signed-off-by: andreibe <95860450+andreibe24@users.noreply.github.com>

* fix: remove duplicate

Signed-off-by: andreibe <95860450+andreibe24@users.noreply.github.com>

* feat: update chart values.yaml

Signed-off-by: andreibe <95860450+andreibe24@users.noreply.github.com>

* feat: update README.md

Signed-off-by: andreibe <95860450+andreibe24@users.noreply.github.com>

* fix: change template behaviour on env

Signed-off-by: andreibe <andreibe24@users.noreply.github.com>

* chore: add helm ci tests

Signed-off-by: andreibe <andreibe24@users.noreply.github.com>

* fix: rerun go test

Signed-off-by: andreibe <andreibe24@users.noreply.github.com>

* test: add extra manifests

Signed-off-by: andreibe <andreibe24@users.noreply.github.com>

* fix: ci tests

Signed-off-by: andreibe <andreibe24@users.noreply.github.com>

---------

Signed-off-by: andreibe <95860450+andreibe24@users.noreply.github.com>
Signed-off-by: andreibe <andreibe24@users.noreply.github.com>
Co-authored-by: andreibe <andreibe24@users.noreply.github.com>
Co-authored-by: Tarun Pothulapati <tarun@dragonflydb.io>
2024-12-02 08:11:50 +00:00
Roman Gershman
c857ff98ad
chore: update command interface for bitops, bloom, zset families (#4235) 2024-12-02 00:25:37 +02:00
Roman Gershman
4a85c69db1
chore: support qlist compression when accounting for memory (#4233)
Also fix "debug objhist" so that its value histogram will show effective malloc
used distributions for all types.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-12-01 21:52:11 +02:00
Kostas Kyrimis
872e49b0b8
fix: command docs (#4229)
Signed-off-by: kostas <kostas@dragonflydb.io>
2024-11-30 16:53:22 +02:00
Roman Gershman
a4b3724929
chore: change the interface of hll, generic and list commands (#4227) 2024-11-29 14:02:51 +02:00
Roman Gershman
3ad5b387bd
chore: Track QList memory (#4226)
After running `debug POPULATE 100 list 100 rand type list elements 10000`
with `--list_experimental_v2=false`:

```
type_used_memory_list:16512800
used_memory:105573120
```

When running with `--list_experimental_v2=true`:

```
used_memory:105573120
type_used_memory_list:103601700
```

TODO: does not yet handle compressed entries correctly but we do not enable compression by default.

Fixes #3800

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-11-29 13:13:23 +02:00
Roman Gershman
68b7baf6a7
chore: qlist code clean ups (#4223)
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-11-29 10:52:01 +02:00
Roman Gershman
183bfaeb67
chore: remove tail field from qlist (#4220)
Also update license's change date.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-11-28 18:27:48 +00:00
Kostas Kyrimis
4aece00aac
chore: add noop docs subcmd (#4214)
* add noop subcommand DOCS to COMMAND
2024-11-28 19:03:04 +02:00
Roman Gershman
010bd8add4
chore: change the interface of stream and server commands (#4219) 2024-11-28 18:44:01 +02:00
Roman Gershman
c9654d4050
chore: Add more qlist tests (#4217)
* chore: Add more qlist tests

Also fix a typo bug in NodeAllowMerge.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>

* chore: fix build

---------

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
2024-11-28 14:54:13 +00:00
Borys
dc04b196d5
test: fix and unskip test_migration_timeout_on_sync (#4216) 2024-11-28 14:54:17 +02:00
adiholden
90b4fea0d9
bug(replication): snapshot cleanup fix in transition to stable sync (#4211)
Signed-off-by: adi_holden <adi@dragonflydb.io>
2024-11-28 08:38:36 +02:00
adiholden
369899a40a chore(helm-chart): update to v1.25.4 2024-11-27 15:39:53 +00:00
Borys
d6f2b76666
fix: cluster_mgr script (#4210) 2024-11-27 14:09:19 +00:00
Roman Gershman
57fd5f16a7
chore: update command interface for hset/set families (#4209) 2024-11-27 14:00:30 +00:00
Stepan Bagritsevich
bd143e4b81
fix(search_family): Fix crash when no SEPARATOR is specified in the FT.CREATE command (#4205)
fix(search_family): fix(search_family): Fix crash when no SEPARATOR is specified in the FT.CREATE command

Signed-off-by: Stepan Bagritsevich <stefan@dragonflydb.io>
2024-11-27 16:30:36 +04:00