* feat: Support ACKs from replica to master
* Rework after CR
* Split the acks into a different fiber and remove the PING loop
* const convention
* move around the order.
* revert sleep removal
* Exit ack fiber on cancellation
* Don't send ACKs if server doesn't support it
Now `SUBSCRIBE` will respond synchronously. The change is here so we:
1. Maintain the order in pipelined requests
2. Don't have a "race condition": subscribe needs to update channel store pointers on all threads. While it awaits for all threads to complete the callback, some of them might have done it earlier, so they can already start sending messages before the initial ack is sent
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
* feat: Use journal LSNs for absolute replication offsets
* 1 - Address small CR comments
2 - Simplify the offset accounting so that we send the correct offset
in `SliceSnapshot::Stop` instead of counting in RdbLoader. This
allows us to revert the changes to slice journaling of EXEC
commands, for example.
* Store int with absl::little_endian
* Document the offset management
Move timeout decrement to the end so we don't assert if we succeed in
the last second (looks like this was the case in the recent regression
test failure on CI)
Clarify the comment of await_synced
Signed-off-by: ashotland <ari@dragonflydb.io>
* test(sentinel_test.py): increase timeout from 10 to 15 seconds in test_failover function
Signed-off-by: ashotland <ari@dragonflydb.io>
---------
Signed-off-by: ashotland <ari@dragonflydb.io>
* feat(server): Save snapshot on shutdown
* CR
* Change save on shutdown to be conditional on --dbfilename.
* Support SHUTDOWN [NO]SAVE and fix unit test
* Better wait for DB loading
* Fix DF format loading state bug
* Fix some fallout from auto save
* feat(server): Insert timestamp into snapshot names explicitly
* Whenever the snapshot filename contains '{timestamp}', it will be substituted with the current local time.
* Default snapshot name is now "dump-{timestamp}"
* InferLoadFile: Modify to recognize "{timestamp}" files correctly.
* ServerFamily::Load: Change extension 'CHECK' into a non-terminating error because it's user-visible
* ServerFamily::DoSave: Add sanity check for the filename extension.
Signed-off-by: Roy Jacobson <roy@dragonflydb.io>
* resolve CR comments
* Add comment about glob sorted output
* Fix InferLoadFile and fix its tests
* Simplify filename behavior with the .dfs format
---------
Signed-off-by: Roy Jacobson <roy@dragonflydb.io>
* fix(regression-tests): Add PortManager
Add PortManager to find and use available ports in regression tests.
Use it in redis_replicaiton_test.
---------
Signed-off-by: ashotland <ari@dragonflydb.io>
fix(regression-test): Delete dump.rdb
This fails redis to start in redis_replicaiton_test,
I assume it was added accidently.
Signed-off-by: ashotland <ari@dragonflydb.io>
* Fix crash in ZPOPMIN
Crash was due to an attempt to access nullptr[-1], which is bad :)
* Add test to repro crash.
There's some leftover debugging statements, they're somewhat useful so I
kept them as the bug is not yet fixed.
* Copy patch by romange to solve the crash
Also re-enable (uncomment) the test in utility.py.
Signed-off-by: chakaz <chakaz@chakaz>
---------
Signed-off-by: chakaz <chakaz@chakaz>
Signed-off-by: Chaka <chakaz@users.noreply.github.com>
Co-authored-by: chakaz <chakaz@chakaz>
1. pytest extensions and fixes - allows running them
with the existing local server by providing its port (--existing <port>).
2. Extend "DEBUG WATCHED" command to provide more information about watched state.
3. Improve debug/vlog printings around the code.
This noisy PR is a preparation before BRPOP fix that will follow later.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Previously we set batch mode when dispatch queue is not empty
but dispatch queue could contain other async messages related to pubsub or monitor.
Now we enable batching only if there are more pipeline commands in the queue.
In addition, fix the issue of unlimited aggregation of batching buffer.
Fixes#935.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
The problem happens when a publisher sends a message and a new subscriber registers.
In that case it sends "subscribe" response and the publish messages and those
interleave sometimes.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* Some test cases for redis replication
Most of them skipped / commented-out to serve as repro without failing
on CI.
---------
Signed-off-by: ashotland <ari@dragonflydb.io>
Store script parameters for each script that allow configuring it's transactions multi mode. They can be configured either for a specific scripts with `SCRIPT CONIFG <sha> [params...]` or changed globally as defaults with `default_lua_config`.
The current supported options are `allow-undeclared-keys` and `disable-atomicity`. Based on those flags, we determine the correct multi mode. `disable-atomicity` allow running in non-atomic mode, whereas being atomic and enabling `allow-undeclared-keys` requires the global mode.
* Ditch docker whcih is complex on CI in favour of local redis binary
Signed-off-by: ashotland <ari@dragonflydb.io>
* Fix typo
Signed-off-by: ashotland <ari@dragonflydb.io>
* Wait for sentinel termination
Signed-off-by: ashotland <ari@dragonflydb.io>
---------
Signed-off-by: ashotland <ari@dragonflydb.io>
* Ditch docker whcih is complex on CI in favour of local redis binary
Signed-off-by: ashotland <ari@dragonflydb.io>
* Fix typo
Signed-off-by: ashotland <ari@dragonflydb.io>
---------
Signed-off-by: ashotland <ari@dragonflydb.io>
* feat(server): Allow admin commands in multi
Needed for sentinel support (#706)
Signed-off-by: ashotland <ari@dragonflydb.io>
* feat(server): Add test coverage for multi global commands
Signed-off-by: ashotland <ari@dragonflydb.io>
* code review fixes
Signed-off-by: ashotland <ari@dragonflydb.io>
* Sentinel integration test
Signed-off-by: ashotland <ari@dragonflydb.io>
* PR code reiew follow up
Have lambda return awaitable instead of defining neoff async function
Signed-off-by: ashotland <ari@dragonflydb.io>
---------
Signed-off-by: ashotland <ari@dragonflydb.io>
* feat(server): Allow admin commands in multi
Needed for sentinel support (#706)
Signed-off-by: ashotland <ari@dragonflydb.io>
* feat(server): Add test coverage for multi global commands
Signed-off-by: ashotland <ari@dragonflydb.io>
* code review fixes
Signed-off-by: ashotland <ari@dragonflydb.io>
* Sentinel integration test
Signed-off-by: ashotland <ari@dragonflydb.io>
---------
Signed-off-by: ashotland <ari@dragonflydb.io>
* test: add dragonfly_db fixture to it tests #199
Signed-off-by: Shmulik Klein <shmulik.klein@gmail.com>
* test: lint using flake8
Signed-off-by: Shmulik Klein <shmulik.klein@gmail.com>
* test: run dragonfly debug version as fallback
Signed-off-by: Shmulik Klein <shmulik.klein@gmail.com>
In rare cases a scheduled transaction is not scheduled correctly and we need
to remove it from the tx-queue in order to re-schedule. When we pull it from tx-queue
and it has been located at the head, we must poll-execute the next txs in the queue.
1. Fix the bug.
2. Improve verbosity loggings to make it easier to follow up on tx flow in release mode.
3. Introduce /txz handler that shows currently pending transactions in the queue.
4. Fix a typo in xdel() function.
5. Add a py-script that reproduces the bug.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Found dangling transaction pointers that where left in the watch queue. Fix the state machine there.
2. Improved transaction code a bit, merged duplicated code into RunInShard function, got rid of RunNoop.
3. Improved BPopper::Run flow.
4. Added 'DEBUG WATCH' command. Also 'DEBUG OBJECT' now returns shard id and the lock status of the object.
