benchmark
benchmark copied to clipboard
pytorch
TorchBench is a collection of open source benchmarks used to evaluate PyTorch performance.
I noticed that `test_bench.py` only supports there devices: `cpu`, `cuda` and `mps`. And I think there should be a power to allow users to run benchmarks on other devices (e.g....
Recently I found that for the same model, the native benchmark code in torchbenchmarks does not give expected time, i.e. one is consistently slower than the other one, or one...
Torch is frequently installed with a local version identifier (e.g. `+cu126`), but that identifier causes an issue with pip because there is no public package with that identifier unless an...
Currently, installation scripts always invoke pip via `sys.executable -m pip` in subprocesses. This PR introduces `get_pip_cmd()`, which: - Uses the `PIP_MODULE` environment variable to override the installer if set. ```bash...
Fix E2E model not working issue, which caused by new API of accelerator and transformers lib Step to reproduce the bug: ```bash python run_e2e.py hf_bert ``` Log ``` Traceback (most...
This adds a benchmark for the Flux image generation pipeline. Specifically, it only benchmarks the diffusion transformer (and omits the text encoder and vae, which don't take up much time...
Right now it's ~1 year behind: https://github.com/pytorch/benchmark/blob/f0b2a09591de5cf276bbfa7ce06a3d35f508da84/torchbenchmark/util/framework/diffusers/requirements.txt#L1 Bumping it would induce breakages like example inputs dimensions in https://github.com/pytorch/benchmark/pull/2654, and maybe more for the old stable diffusion models. Maybe we can...
when running `python benchmarks/dynamo/torchbench.py --accuracy --inference --bfloat16 --device cpu --inductor --only Background_Matting`，the following error occurs: If I run `python install.py Background_Matting` to get model data, the following error occurs: How...
fix bug in `pytorch 2.8.0` `ModuleNotFoundError: No module named 'torch._C._nativert'; 'torch._C' is not a package` * command ``` TORCHINDUCTOR_UNIQUE_KERNEL_NAMES=1 TORCHINDUCTOR_BENCHMARK_KERNEL=1 python -u userbenchmark/dynamo/dynamobench/torchbench.py --backend inductor --float16 --performance --only BERT_pytorch --training...
