Python wrapper for text-generation-benchmark utility

[antferdom]

Feature request

Control the benchmarking utility output, format and location, for running the benchmarking using system Python wrappers as subprocess or pexpect. Writing the output to a file might be sufficient.

Hypothetical example:

ret = subprocess.run(["text-generation-benchmark",
                                        "--batch-size", "1",
                                        "--sequence-length", "10",
                                        "--decode-length", "10",
                                        "--tokenizer-name", "bigscience/mt0-xxl-mt",
                                      ],
                                     universal_newlines=True,)

Motivation

The benchmark utility spawns a TUI application, which makes the interactive experimentation great, but complicates the automation of benchmarking. The app is inspired by oha, but this implementation does not allow to specify any flag for disable realtime TUI, like Oha's --no-tui.

Your contribution

Not contribution staged yet.

[OlivierDehaene]

I'm a bit confused. Do you want a wrapper or do you want the --no-tui option to exist?

[antferdom]

Sorry for the confusing explanation. If we add the option of non-interactive TUI, it might be sufficient for interacting with the application and its output using subprocess as a Python wrapper. Another way around is to pass output location and output format to the text-generation-benchmark, and then simply consume that from the language of choice.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.