* chore: preparation for basic http api
The goal is to provide very basic support for simple commands,
fancy stuff like pipelining, blocking commands won't work.
1. Added optional registration for /api handler.
2. Implemented parsing of post body.
3. Added basic formatting routine for the response. It does not cover all the commands but should suffice for
basic usage.
The API is a POST method and the body of the request should contain command arguments formatted as json array.
For example, `'["set", "foo", "bar", "ex", "100"]'`.
The response is a json object with either `result` field holding the response of the command or
`error` field containing the error message sent by the server.
See `test_http` test in tests/dragonfly/connection_test.py for more details.
* chore: cover iouring with enable_direct_fd
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* fix(helm): add issuer group to create the certificate without wait for the previous created issuer
Signed-off-by: Fabiano Arruda Ferreira das Graças <fafg@fafg-mbm1.fritz.box>
* fix(helm): remove condition that can prevent the helm chart be rendered on machines where monitoring.coreos.com is not installed or is not the end target of the helm template command
Signed-off-by: fafg <fabiano.arruda@hotmail.com>
* fix(helm): lint - remove blank line
Signed-off-by: fafg <fabiano.arruda@hotmail.com>
* add(helm): missing service monitor test files
Signed-off-by: fafg <fabiano.arruda@hotmail.com>
* add(helm): add missing cert-manager test files
Signed-off-by: fafg <fabiano.arruda@hotmail.com>
* fix(helm): lint - add missing blank lines
Signed-off-by: fafg <fabiano.arruda@hotmail.com>
* fix(helm): rebase
Signed-off-by: fafg <fabiano.arruda@hotmail.com>
* Revert "fix(helm): rebase"
This reverts commit c4ce16b76e.
* fix(helm): fix service monitor namespace rendering
Signed-off-by: fafg <fabiano.arruda@hotmail.com>
* fix(helm): add missing up to date golden file
Signed-off-by: fafg <fabiano.arruda@hotmail.com>
* fix(helm): merge upstream
Signed-off-by: fafg <fabiano.arruda@hotmail.com>
* update golden files
* also install prom operator dependencies
* also install cert-manager
* skip cert-manager chart
* skip cert-manager value
* remove CI TLS files
* fix formatting
* fix formatting
* fix actions
---------
Signed-off-by: Fabiano Arruda Ferreira das Graças <fafg@fafg-mbm1.fritz.box>
Signed-off-by: fafg <fabiano.arruda@hotmail.com>
Co-authored-by: Tarun Pothulapati <tarun@dragonflydb.io>
Co-authored-by: Tarun Pothulapati <tarunpothulapati@outlook.com>
* feat(replication): Do not auto replicate different master
Until now, replicas would re-connect and re-replicate a master after the
master will restart. This is problematic in case the master loses its
data, which will cause the replica to flush all and lose its data as
well.
This is a breaking change though, in that whoever controls the replica
now has to explicitly issue a `REPLICAOF X Y` in order to re-establish
a connection to a new master. This is true even if the master loaded an
up to date RDB file.
It's not necessary if the replica lost connection to the master and the
master was always alive, and the connection is re-established.
Fixes#2636
* fix test
* fixes
* proxy proxy java java
* better comment
* fix comments
* replica_reconnect_on_master_restart
* proxy.close()
1. More test fixes to support both types
2. Adjust Aggregation functions to work with both types
3. jsonpath_test build from both types and passes for JsonType,
the tests are skipped for flexbuffers
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Also adjust jsonpath_test to be a typed test to allow run the same tests
for both jsoncons and flatbuffers.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Use clib malloc for allocating fiber stacks but reduce the fiber stack size.
clib malloc uses default 4K OS pages when reserving memory from the OS.
The reason for not using mi_malloc, because we use 2MB large OS pages with mimalloc.
However, allocating stacks is one of the cases, when using smaller 4KB memory pages is actually more
RSS efficient because memory pages become hot at better granularity.
2. Add "memory_fiberstack_vms_bytes" metric exposing fiber stack vm usage.
3. Fix macos dependencies & update ci versions.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* feat(json): Deserialize ReJSON format
This PR adds support for Redis-based JSON RDB format deserialization.
Since Redis uses ReJSON as a module, serialization is slightly different
from other types, but overall it's not a big change once we know where
all bits should be.
While this change knows how to _read_ Redis-based JSON keys, it does not
_save_ them in Redis format. That will be in a different PR.
This PR also ignores unknown (non-keys) module data instead of failing the load.
Fixes#2718
* Cleanup
* Add tests
* Skip unsupported modules
* Small refactor
1. Replaces run_barrier as a synchronization point with is_armed + an embedded blocking counter for awaiting running jobs
2. Replaces IsArmedInShard + GetLocalMask + is_armed.exchange chain with a single DisarmInShard() / DisarmInShardWhen
Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
1. Pipeline squashing was not recorded
2. Apparently Redis counts commands of MULTI/EXEC transations separately, so I assume we also should
-> Place RecordCmd() in Invoke()
* chore: prevent crashing upon inconsistent expiry table
Also, introduce "DFLY LOAD <filename>" command in addition to "DEBUG LOAD"
as an official command to load snapshots into the running server.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
We already have a Fiber-aware DNS resolver in Helio, so it's trivial to
change and use.
I tested this end-to-end and it really resolves DNS addresses, not just
localhost.
Fixes#947
* chore: improve compatibility of EXPIRE functions with Redis
Also, provide a module name if stumbled upon module data that can not be loaded
by dragonfly.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* feat(server): Use mimalloc in SSL calls
Until now, OpenSSL used `malloc()` directly. This PR overrides it to use
mimalloc.
Fixes#2709
* Add generate-tls-files.sh
* feat(cluster): Add `--cluster_id` flag
This flag sets the unique ID of a node in a cluster.
It is UB (and bad) to set the same IDs to multiple nodes in the same
cluster.
If unset (default), the `master_replid` (previously known as `master_id`) is used.
Fixes#2643
Related to #2636
* gh comments
* oops - revert line removal
* fix
* replica
* disallow cluster_node_id in emulated mode
* fix replica test
* chore: add malloc-based stats and decommit
Provides more stats and control with glibc-malloc based allocator.
For example,
with v1.15.0 (--proactor_threads=2), empty database, `info memory`returns
```
used_memory:614576
used_memory_human:600.2KiB
used_memory_peak:614576
used_memory_peak_human:600.2KiB
used_memory_rss:19922944
used_memory_rss_human:19.00MiB
```
then during `memtier_benchmark -n 300000 --key-maximum 100000 --ratio 0:1 --threads=30 -c 100` (i.e GET-only with 3k connections):
```
used_memory:614576
used_memory_human:600.2KiB
used_memory_peak:614576
used_memory_peak_human:600.2KiB
used_memory_rss:59985920
used_memory_rss_human:57.21MiB
used_memory_peak_rss:59985920
```
connections overhead grows by ~39MB.
when the traffic stops, `used_memory_rss_human` becomes `30.35MiB`
and we do not know where 11MB gets lost and `MEMORY DECOMMIT` does not reduce the RSS.
With this change, `memory malloc-stats` return during the memtier traffic
```
malloc arena: 394862592
malloc fordblks: 94192
```
i.e. 395MB virtual memory was allocated by malloc and only 94KB is chunks available for reuse.
395MB is arena virtual memory, and not RSS obviously, but at least we have some visibility into malloc reservations.
The RSS usage is the same ~57MB and the difference between virtual and RSS is due to the fact we reserve fiber stacks of size 131KB but we touch less.
After the traffic stops, `arena` is reduced to 134520832 bytes, and fordblks are 133016592, i.e. majority of reserved ranges are also free (available to reuse) in the malloc pools.
RSS goes down similarly to before to ~31MB.
So far, this PR only demonstrated the increased visibility to mmapped ranges reserved by glibc malloc.
The additional functional change is in `MEMORY DECOMMIT` that now trims malloc RSS usage from reserved but unused (fordblks) pages
by calling `malloc_trim`.
After the call, RSS is: `used_memory_rss_human:20.29MiB` which is almost the same as when we started the empty process.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: fix build for older glibc environments
Disable these extensions for alpine and use legacy version
for older glibc libraries.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
