Improve the autoscaling benchmark scripts

[Jeffwan]

🚀 Feature Description and Motivation

Copy some TODO items from issue https://github.com/aibrix/aibrix/pull/650

WIP items:

• [ ] Improving aibrix/benchmarks/generator/client.py with async and streaming mode
  • [ ] TTFT
  • [ ] TPOT
  • [ ] Goodput
• [ ] Adjust the QPS scale
• [ ] Improving plot script
  • [ ] Tokens/s times series plot
  • [ ] TTFT times series plot
  • [ ] TPOT times series plot
  • [ ] Goodput times series plot

Use Case

This is to improve the qualify of the autoscaling experiments

Proposed Solution

No response

[happyandslow]

Addressing in https://github.com/aibrix/aibrix/pull/676

[Jeffwan]

@happyandslow what's the status of this story?

1. All above metrics in the issue description are all collected?
2. Did you make changes to the plot diagram?
3. is #667 resolved?

[Jeffwan]

@happyandslow Any updates?

[happyandslow]

@happyandslow Any updates?

The modification is applied to #676 Improving aibrix/benchmarks/generator/client.py with async and streaming mode

• [x] TTFT
• [x] TPOT
• [x] Goodput
• [x] Adjust the QPS scale

Improving plot script

• [x] Tokens/s times series plot
• [x] TTFT times series plot
• [x] TPOT times series plot
• [ ] Goodput times series plot

I'm leaving out Goodput since it is a one time measurement throughout the entire experiment.

I'm waiting for some free cycles to test these out before PR can be merged.

[Jeffwan]

I'm waiting for some free cycles to test these out before PR can be merged.

@happyandslow

Do you mean the changes has been included in #676 or in coming PR?

Can you confirm whether #667 is resolved or not?

[happyandslow]

#667 should solved (check comments) The changes I was going to try were #862

[Jeffwan]

Great. We can close this issue, we can revisit #862 once it's ready.