* chore: get back on the decision to put a hard limit on command interface
Limiting commands to only Transaction* and SinkReplyBuilder does not hold.
We need sometimes to access context fields for multitude of reasons.
But I do not want to pass the huge ConnectionContext object because, it's hard
then to track unusual access patterns.
The compromise: to introduce CommandContext that currently has tx, rb and extended fields.
It will be relatively easy to identify irregular access patterns by tracking the extended field.
This commit is the first one in series of probably 10-15 commits. No functional changes here.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Should work only for self-hosted runners.
The core files will be kept in /var/crash/
We also copy automatically dragonfly binary into /var/crash to be able to debug later.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Dragonfly responds to ascii based requests to tls port with:
`-ERR Bad TLS header, double check if you enabled TLS for your client.`
Therefore, it is possible to test now both tls and non-tls ports with a plain-text PING.
Fixes#4171
Also, blacklist the bloom-filter test that Dragonfly does not support yet.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* fix: deduplicate mget response
In case of duplicate mget keys, skips fetching the same key twice.
The optimization is straighforward - we just copy the response for the original key,
since the response is a shallow object, we potentially save lots of memory with this
deduplication. Always deduplicate inside OpMGet.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* feat: Huge values breakdown in cluster migration
Before this PR we used `RESTORE` commands for transferring data between
source and target nodes in cluster slots migration.
While this _works_, it has a side effect of consuming 2x memory for huge
values (i.e. if a single key's value takes 10gb, serializing it will
take 20gb or even 30gb).
With this PR we break down huge keys into multiple commands (`RPUSH`,
`HSET`, etc), respecting the existing `--serialization_max_chunk_size`
flag.
Part of #4100
The long-term goal is to make the parser to consume the whole input when
it returns INPUT_PENDING. It requires several baby step PRs.
This PR:
1. Adds more invariant checks
2. Avoids calling RedisParser::Parse with an empty buffer.
3. In bulk string parsing - remove redundant "optimization" of rejecting partial strings of less than 32 bytes,
in other words consume small parts as well. The unit test adjusted accordingly.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Update dragonfly.logrotate
If multiple logs are being rotated and one of them fails (due to exit 1), the other logs that follow won't be rotated either, unless logrotate is run again.
If you want to prevent the rotation of a specific log file and not affect the rest of the logs, you'll want to handle the condition properly to ensure that logrotate doesn't abort due to the failure of the prerotate script.
To prevent the rotation of a specific log file without causing issues for other logs, you can use exit 0 to prevent rotation cleanly or design your prerotate script to handle conditions carefully.
Signed-off-by: s13k <s13k@pm.me>
1. Better logging in regtests
2. Release resources in dfly_main in more controlled manner.
3. Switch to ignoring signals when unregister signal handlers during the shutdown.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* fix: Huge entries fail to load outside RDB / replication
We have an internal utility tool that we use to deserialize values in
some use cases:
* `RESTORE`
* Cluster slot migration
* `RENAME`, if the source and target shards are different
We [recently](https://github.com/dragonflydb/dragonfly/issues/3760)
changed this area of the code, which caused this regression as it only
handled RDB / replication streams.
Fixes#4143
Fixes#4150. The failure can be reproduced with high probability on ARM via
`pytest dragonfly/replication_test.py -k test_replication_all[df_factory0-mode0-8-t_replicas3-seeder_config3-2000-False]`
Not sure why this barrier is needed but #4146 removes the barrier
which breaks a gentle balance in the code in unexpected way.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* fix: enforce load limits when loading snapshot
Prevent loading snapshots with used memory higher than max memory limit.
1. Store the used memory metadata only inside the summary file
2. Load the summary file before loading anything else, and if the used-memory is higher,
abort the load.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
1. Now error stats show "restrict_denied" instead of "Cannot execute restricted command ..." error.
2. Increased verbosity level when loading a key with expired timestamp.
3. pulled helio with better logs coverage of tls_socket.cc code.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* chore: optimize info command
Info command has a large latency when returning all the sections.
But often a single section is required. Specifically,
SERVER and REPLICATION sections are often fetched by clients
or management components.
This PR:
1. Removes any hops for "INFO SERVER" command.
2. Removes some redundant stats.
3. Prints latency stats around GetMetrics command if it took to much.
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* Update src/server/server_family.cc
Co-authored-by: Shahar Mike <chakaz@users.noreply.github.com>
Signed-off-by: Roman Gershman <romange@gmail.com>
* chore: remove GetMetrics dependency from the REPLICATION section
Also, address comments
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
* fix: clang build
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
Signed-off-by: Roman Gershman <romange@gmail.com>
Co-authored-by: Shahar Mike <chakaz@users.noreply.github.com>
**The problem:**
When in cluster mode, `MOVED` replies (which are arguably not even errors) are aggregated per slot-id + remote host, and displayed in `# Errorstats` as such. For example, in a server that does _not_ own 8k slots, we will aggregate 8k different errors, and their counts (in memory).
This slows down all `INFO` replies, takes a lot of memory, and also makes `INFO` replies very long.
**The fix:**
Use `type` `MOVED` for moved replies, making them all the same under `# Errorstats`
Fixes#4118
* chore: simplify BumpUps deduplication
This pr #2474 introduced iterator protection by
tracking which keys where bumped up during the transaction operation.
This was done by managing keys view set. However, this can be simplified
using fingerprints. Also, fingerprints do not require that the original keys exist.
In addition, this #3241 PR introduces FetchedItemsRestorer that tracks bumped set and
saves it to protect against fiber context switch. My claim is that it's redundant.
Since we only keep the auto-laundering iterators, when a fiber preempts these iterators recognize it
(see IteratorT::LaunderIfNeeded) and refresh themselves anyway.
To summarize: fetched_items_ protect us from iterator invalidation during atomic scenarios,
and auto-laundering protects us from everything else, so fetched_items_ can be cleared in that case.
---------
Signed-off-by: Roman Gershman <roman@dragonflydb.io>
