AutoMQ
[Good First Issue] Add Support for Specifying Topic Prefix for Catch-up Reading in Performance Test
Background
AutoMQ provides the automq-perf-test.sh script to test performance under different scenarios. Currently, it supports testing "catch-up read" (cold read) scenarios by:
- Sending a batch of messages.
- Delaying consumption until messages have accumulated for a specified period.
- Observing consumption throughput and its impact on producers.
But currently, each test requires spending significant time to accumulate messages, which slows down the test.
Goal
Save testing time by allowing users to pre-specify topics or topic groups (via a prefix) for catch-up read testing.
Required Changes
- Add a parameter (e.g.,
--catchup-topic-prefix) to specify a topic prefix for catch-up read testing. - If the prefix is provided, reuse existing topics matching the prefix instead of creating new ones.
- Ensure the script skips the message accumulation phase when using existing topics.
Hello, I have a few questions in the process of implementing this issue, and I hope to get answers. Regarding your optimization requirement: "By adding the -catchup-topic-prefix prefix, the script can reuse the existing topic", I found that the perfcommend class is the actual test tool. Do I need to modify the code of this class to verify the "-catchup-topic-prefix" prefix parameter, or can I meet the requirement by simply modifying the logic of automq-perf-test.sh?
Hello, I have a few questions in the process of implementing this issue, and I hope to get answers. Regarding your optimization requirement: "By adding the -catchup-topic-prefix prefix, the script can reuse the existing topic", I found that the perfcommend class is the actual test tool. Do I need to modify the code of this class to verify the "-catchup-topic-prefix" prefix parameter, or can I meet the requirement by simply modifying the logic of automq-perf-test.sh?
Hello, @xiayu1118, the automq-perf-test.sh script primarily serves as an entry point that ultimately executes the org.apache.kafka.tools.automq.PerfCommand#main method. To implement this feature, you will need to modify the relevant Java code rather than just the shell script logic.
Specifically, you should add the new command-line option (e.g., --catchup-topic-prefix) in the org.apache.kafka.tools.automq.perf.PerfConfig class to parse and validate the parameter. Then, implement the corresponding logic in the org.apache.kafka.tools.automq.PerfCommand and org.apache.kafka.tools.automq.perf.ConsumerService class to handle topic reuse based on the prefix.
Feel free to reach out if you need further clarification or assistance during implementation!
Hello, I have a few questions in the process of implementing this issue, and I hope to get answers. Regarding your optimization requirement: "By adding the -catchup-topic-prefix prefix, the script can reuse the existing topic", I found that the perfcommend class is the actual test tool. Do I need to modify the code of this class to verify the "-catchup-topic-prefix" prefix parameter, or can I meet the requirement by simply modifying the logic of automq-perf-test.sh?
Hello, @xiayu1118, the
automq-perf-test.shscript primarily serves as an entry point that ultimately executes theorg.apache.kafka.tools.automq.PerfCommand#mainmethod. To implement this feature, you will need to modify the relevant Java code rather than just the shell script logic.Specifically, you should add the new command-line option (e.g.,
--catchup-topic-prefix) in theorg.apache.kafka.tools.automq.perf.PerfConfigclass to parse and validate the parameter. Then, implement the corresponding logic in theorg.apache.kafka.tools.automq.PerfCommandandorg.apache.kafka.tools.automq.perf.ConsumerServiceclass to handle topic reuse based on the prefix.Feel free to reach out if you need further clarification or assistance during implementation!
Got it！
Hello, I encountered a problem while trying to test my new bash script that supports reusing existing themes through prefixes: Does autoMQ currently support source code deployment? I encountered a lot of problems while compiling and packaging autoMQ, and I was wondering if I had gone in the wrong direction; Or is there a more convenient way to run test scripts?
