benchmark icon indicating copy to clipboard operation
benchmark copied to clipboard

Torchbench models that don't run in dynamo runners

Open msaroufim opened this issue 1 year ago • 1 comments

There's small nuances in how the dynamo runners benchmark models that can make certain torchbench models fail

Some models might be explicitly skipped, others might fail because of some dtype conversion. This can be frustrating because if you add a model to torchbench like clip or cm3leon you won't see it in the pt2dashboard so creating this giant tracker issue to solve this

To repro: look at logs in HUD for e.g https://ossci-raw-job-status.s3.amazonaws.com/log/16535270177 and compare to model names in models/ and canary_models/

If something is showing up in unique to torchbench that means it's not showing up in the pt2 dashboard

There are some concrete things we could do better in dynamo runners like starting with loudly erroring but should also track what these failures are

Notably I found this problem out after investigating stable diffusion and cm3leon

msaroufim avatar Sep 15 '23 20:09 msaroufim