optimum-benchmark
optimum-benchmark copied to clipboard
huggingface
A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations & quantization schemes.
Execute benchmark with Hydra CLI, Get Errors below. The config yaml is attached as .txt user@06d8461fae8f:/optimum-benchmark$ optimum-benchmark --config-dir ./examples --config-name diffusion_cuda.yaml [MAIN-PROCESS][2024-05-16 15:20:19,735][backend][WARNING] - `device_ids` was not specified, using all...
- set training_arguments.use_cpu to true so that the Trainer class runs training on the cpu instead of utilizing the available gpu
The training benchmark link no longer works : https://huggingface.co/blog/huggingface-and-optimum-amd How can one test training throughput on AMD these days? Also, can you provide details about the experiments in the figure...
We need an api that allows: - loading `benchmark_report.json` files into `Report` instances - detect regressions between two `Report` instances
Hello, I'm running base mistral and using bnb to quantize as below: ```python from optimum_benchmark.backends.pytorch.config import PyTorchConfig from optimum_benchmark.benchmarks.inference.config import InferenceConfig from optimum_benchmark.experiment import ExperimentConfig, launch from optimum_benchmark.launchers.process.config import ProcessConfig...
I ran tests cpu_training_pytorch_bert_sweep and cpu_training_pytorch_gpt_sweep on a machine with cpu / gpu enabled. Those tests utilize gpu for training if it is available. steps to reproduce: - install optimum-benchmark...
Can I use Optimum-benchmark to evaluate the performance of qwen.cpp or llama.cpp？
@regisss would it make sense to add task specific evaluators. for example with `automatic-speech-recognition`, as I did it manually when I did whisper's benchmark.
Metadata
Owner
Metadata
A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations & quantization schemes.