* chore: Introduce pipeline back-pressure
Also, improve synchronization primitives and replace them with
thread-local variations.
Before the change, on my local machine with the dragonfly running with 8 threads,
`memtier_benchmark -c 10 --threads 8 --command="PING" --key-maximum 100000000 --hide-histogram --distinct-client-seed --pipeline=20 --test-time=10`
reached 10M qps with 0.327ms p99.9.
After the change, the same command showed 13.8M qps with 0.2ms p99.9
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: export pipeline related metrics
Export in /metrics
1. Total pipeline queue length
2. Total pipeline commands
3. Total pipelined duration
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* fix(helm): add issuer group to create the certificate without wait for the previous created issuer
Signed-off-by: Fabiano Arruda Ferreira das Graças <fafg@fafg-mbm1.fritz.box>
* fix(helm): remove condition that can prevent the helm chart be rendered on machines where monitoring.coreos.com is not installed or is not the end target of the helm template command
Signed-off-by: fafg <fabiano.arruda@hotmail.com>
* fix(helm): lint - remove blank line
Signed-off-by: fafg <fabiano.arruda@hotmail.com>
* add(helm): missing service monitor test files
Signed-off-by: fafg <fabiano.arruda@hotmail.com>
* add(helm): add missing cert-manager test files
Signed-off-by: fafg <fabiano.arruda@hotmail.com>
* fix(helm): lint - add missing blank lines
Signed-off-by: fafg <fabiano.arruda@hotmail.com>
* fix(helm): rebase
Signed-off-by: fafg <fabiano.arruda@hotmail.com>
* Revert "fix(helm): rebase"
This reverts commit c4ce16b76e.
* fix(helm): fix service monitor namespace rendering
Signed-off-by: fafg <fabiano.arruda@hotmail.com>
* fix(helm): add missing up to date golden file
Signed-off-by: fafg <fabiano.arruda@hotmail.com>
* fix(helm): merge upstream
Signed-off-by: fafg <fabiano.arruda@hotmail.com>
* update golden files
* also install prom operator dependencies
* also install cert-manager
* skip cert-manager chart
* skip cert-manager value
* remove CI TLS files
* fix formatting
* fix formatting
* fix actions
---------
Signed-off-by: Fabiano Arruda Ferreira das Graças <fafg@fafg-mbm1.fritz.box>
Signed-off-by: fafg <fabiano.arruda@hotmail.com>
Co-authored-by: Tarun Pothulapati <tarun@dragonflydb.io>
Co-authored-by: Tarun Pothulapati <tarunpothulapati@outlook.com>
1. Use clib malloc for allocating fiber stacks but reduce the fiber stack size.
clib malloc uses default 4K OS pages when reserving memory from the OS.
The reason for not using mi_malloc, because we use 2MB large OS pages with mimalloc.
However, allocating stacks is one of the cases, when using smaller 4KB memory pages is actually more
RSS efficient because memory pages become hot at better granularity.
2. Add "memory_fiberstack_vms_bytes" metric exposing fiber stack vm usage.
3. Fix macos dependencies & update ci versions.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Also update actions versions to Node 20.
This change allows dragonfly to be built on MacOs.
However, we still have multiple failing tests on MacOS.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* upload only failed test logs
* remove printing log names for passed tests
* print slow tests with --duration
* separate regression and unit logs for CI workflow
* test(memory): Test memory accounting for all types
* slightly faster
* WIP
* working
* Document
* Update test to use DEBUG POPULATE
* Nothing much
* Working
* fix
* yaml
* explicit capture
* fix ci?
* stub tx
chore: UpdateHelio dependency
Add support for asan/ubsan checkers in our dev environment.
Remove more clang warnings.
Once we fix all the problems we will enable them in our CI as well.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
This is needed if we want to allow asynchronous transactional operations during the callback execution.
Also update actions versions.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
We had a place in tools/packaging/generate_debian_package.sh that relied on the existence of build-opt,
moreover, if it did not exist the script deadlocked.
1. Added more loggings
2. Removed the loop
3. Removed unnecessary dependency in the script on the build-dir name.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: fix our release pipeline
Also remove alpine prod.wip file that has not been used and unlikely will be for prod.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Also, change its build directory to build-release.
Simplify a bit its configuration steps as well. No change in functionality is expected.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
chore: run dragonfly_test with epoll under gdb
Also, update helio that provide a stacktrace under musl libc (alpine linux).
This version of helio updates absl version as well.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* add a python script to print the most recent log
* if CI timeouts, print the most recent log
* replace global timeout with timeout command
* upload all logs on failure()
* print uid + port + the log files for each df instance
* chore(regression): test bptree on regression pytests
1. stop passing the flag use_zset_tree as it is true on default
2. fix ci test to run replication tests
3. change replication tests seeder to sometimes add more than 128 values
to zset to test the pbtree impl
Signed-off-by: adi_holden <adi@dragonflydb.io>
1. Move docker build files to separate dir from docker script files
so that they won't be part of build context. Update dockerignore as well
2. Fix lib dependencies for alpine
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
fix: use "cd" instead of woarking-directory that does not work
Also, use GITHUB_WORKSPACE due to https://github.com/actions/runner/issues/2058
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* Add a coverage workflow to dragonfly.
* Try and make the testing permissive
* Apply suggestions from code review
Signed-off-by: Roy Jacobson <roi.jacobson1@gmail.com>
* Remove redundant parts
---------
Signed-off-by: Roy Jacobson <roi.jacobson1@gmail.com>
* test(server): test transaction locked keys
1. add test utility class that will add suspension to transaction
execution.
2. add test for locked keys in transaction
Signed-off-by: adi_holden <adi@dragonflydb.io>
* feat(server): Add `--lock_on_hashtags` mode.
This new mode effectively locks hashtags (i.e. strings within {curly
braces}) instead of the full keys being used.
This can allow scripts to access undeclared keys if they all use a
common hashtag, like for the case of BullMQ.
To make sure this mode is tested, I added a way to specify flags via env
variables, and modified `ci.yml` to run all tests using this mode as well.
While at it, I also added `--cluster_mode=emulated` mode to CI.
* The bug - if all commands inside multi trasaction are eval commands
and global scripts mode is on, we did ignored the trasaction
and run each eval separately.
*Fix - run all evals under multi inside the global lock
* Change multi eval run only if scripts are in global mode and multi
mode is not non atomic
* Fix test flags setup
* Skip test ContendedList as it fails
* change default exec mode for txs to lock ahead
Currently, During docker release we don't actually build the
alpine release but the new docker run test ends up trying to
run it and fails. This adds the same toggle that we use for
build to prevent the test step.
This commit updates the weekly docker build to use `dragonfly-weekly`
image tag so that we get better separation. We also now push these
images with the github sha commit tags. We also update `latest` as
these get pushed.
It's a bit more efficient than Boost.Fibers due to better integrations
of Fibers with Proactor loop.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
