apache
HDDS-11542. Add a new freon subcommand to read random keys
What changes were proposed in this pull request?
Background: to measure memory consumption of Ozone, it is important to have a tool to create solely read workload. Existing freon subcommands are not suitable for this case.
I add a new freon subcommand that selects random volume/bucket pairs and read keys from them.
Example command: ozone freon rkr --duration=10m
This example reads keys from random volume/bucket pairs for 10 minutes.
At the end it reports the number of keys read and timer report.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-11542
How was this patch tested?
Tested locally.
I agree with @adoroszlai comment, as we already have freon commands which create random keys for a particular bucket, having a similar command for reading keys from 1 bucket would be more useful than a random read.
Let me add some background on why I proposed to @len548 to create this tool.
In freon we have the possibility to generate key ranges, we also have the possibility to generate range keys and the read/list those ranges within the same workload. These are good tools, but they are working in a predictable manner on key ranges.
We are curious to see how Ozone behaves and utilizes memory and also we are interesting in how Ozone behaves if there are frequent cache misses on the RocksDB level or the on heap partial cache for key related data. (afaik we have some caching on heap for keys also, let me know if I am wrong)
This is not something you can achieve with predictable key ranges.
So this tool is the reader counterpart of the random key generator, and reads random keys. If the two are used together, then the results are predictable, as the random key generator generates keys of the same size, and if we read any of the keys we read the same amount of data from a datanode, while if we read it with this level of randomness, then my prediction is that (given enough keys), there will be a lot of cache misses within OM, and within the DNs we also do not read data that were written at around the same time, so OS caches for I/O will also have misses.
I believe this gives us more insight into a real workload where potentially a lot of clients read different data at scale, and will be able to give us insight into what happens with memory consumption and performance in a highly utilized cluster where potentially any data can be read by multiple clients at the same time.
Please correct me if I am wrong, but as I understand, with other existing tools this level of read unpredictability can not be reached.
On a side note, this tool is not really give any insight to anything if the key sizes are different, so it is not meaningful to use this on a real cluster with varying file sizes in a real environment. So @adoroszlai I completely agree with you, this tool is not producing useful info in a real env, but combined with random key generator I believe the unpredictability of the results is something that gives insight and useful. What do you think?
we also have the possibility to generate range keys and the read/list those ranges within the same workload
ozone freon ockv is a read-only workload, which reads a range of keys written at any time. It is the read counterpart of ockg, and expects keys to be in the same prefix/counter format.
Yes, I meant, similar to ockv and ockg, this tool can be the counterpart of freon's randomkeys tool, which writes same sized random keys with random names, and this tool can read them.
Compared to ockv the random key read can be used to read keys that are there for any other reason, but that is not the intended use case and it does not really give real insights.
As a counterpart of randomkeys, it seems to be useful, isn't it? It may be we should restrict the list of buckets it reads from, so that it is measuring reads from certain buckets, and with that it would be usable on clusters with arbitrary data pre-exists, as that one is missing at the moment.
We came up with the idea regarding to a test in a fresh environment, where we have generated a bunch of random keys, and we did not wanted to re-generate the data as ranges of keys so existing data and co-existing data was not considered.
It may be we should restrict the list of buckets it reads from, so that it is measuring reads from certain buckets, and with that it would be usable on clusters with arbitrary data pre-exists, as that one is missing at the moment.
Agree, that's kind of what I suggested earlier:
I think a simple tool to read all keys of a given bucket would be more useful.
It can be a list of buckets instead of just one.
Thank you very much for the patch. I am closing this PR temporarily as there was no activity recently and it is waiting for response from its author.
It doesn't mean that this PR is not important or ignored: feel free to reopen the PR at any time.
It only means that attention of committers is not required. We prefer to keep the review queue clean. This ensures PRs in need of review are more visible, which results in faster feedback for all PRs.
If you need ANY help to finish this PR, please contact the community on the mailing list or the slack channel."
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}