4e3be726c5
* chore(pytest): Refactor snapshot test --------- Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>
2.5 KiB
Seeder library
Please use the testing frameworks factories to obtain proper seeder instances!
1. Static seeder
The StaticSeeder is a thin wrapper around DEBUG POPULATE with a little bit of fuzziness for collection sizes. It should be preffered for generating "static" data for snapshotting, memory consumption tests, etc.
s = StaticSeeder(key_target=10_000)
await s.run(client) # Creates around 10k keys
2. Checking consistency
Use SeederBase.capture() (accessed via StaticSeeder or Seeder) to calculate a "state hashes" based on all the data inside an instance. Equal data produces equal hashes (equal hashes don't guarantee equal data but what are the odds...).
# Fill master with ~10k keys
s = StaticSeeder(key_target=10_000)
await seeder.run(master)
# "Replicate" or other operations
replicate(master, replica)
# Ensure master and replica have same state hashes
master_hashes, replica_hashes = await asyncio.gather(
StaticSeeder.capture(master), # note it's a static method
StaticSeeder.capture(replica)
)
assert master_hashes == replica_hashes
3. Dynamic seeder
Contrary to the static seeder, the normal seeder issues a more complicated mix of commands, supports deleting keys and sending modification traffic. The seeder tries to maintain a specific number of keys, quickly filling or emptying the instance to reach the target. Once reached, it will issue a balanced load of all kinds of operations.
# Configure how many keys we want
s = Seeder(key_target=10_000)
# Fill instance with keys until it's 10k +- 1%
# Will create many new keys with data and reach equilibrium
await s.run(client, target_deviation=0.01)
assert abs(client.dbsize() - 10_000) <= 100
# Run 5k operations, balanced mix of create/delete/modify
await s.run(client, target_ops=5000)
# Now we want only 500 keys, issue many deletes
s.change_key_target(500)
await s.run(client, target_deviation=0.01)
4. Working with load
A seeders run(client) can be called without any target. It can only be stopped with
# Fill instance with keys
s = Seeder()
await seeder.run(client, target_deviation=0.01)
# Start seeder without target
# Because the instance reached its key target, the seeder
# will issue a balanced mix of modifications/additions/deletions
seeding_task = asyncio.create_task(s.run(client))
# Do operations under fuzzy load
save(client)
await s.stop(client) # request stop, no immediate effect
await seeding_task # wait for actual stop and cleanup