Ean Garvey

``` FAILED tank/facebook_convnext-tiny-224_tf/facebook_convnext-tiny-224_tf_test.py::ConvNextTinyModuleTest::test_module_dynamic_gpu FAILED tank/facebook_convnext-tiny-224_tf/facebook_convnext-tiny-224_tf_test.py::ConvNextTinyModuleTest::test_module_static_gpu FAILED tank/facebook_deit-small-distilled-patch16-224_torch/facebook_deit-small-distilled-patch16-224_torch_test.py::DeitModuleTest::test_module_static_gpu FAILED tank/google_vit-base-patch16-224_tf/google_vit-base-patch16-224_tf_test.py::VitBaseModuleTest::test_module_dynamic_gpu FAILED tank/google_vit-base-patch16-224_tf/google_vit-base-patch16-224_tf_test.py::VitBaseModuleTest::test_module_static_gpu FAILED tank/google_vit-base-patch16-224_torch/google_vit-base-patch16-224_torch_test.py::VitBaseModuleTest::test_module_static_gpu FAILED tank/nvidia_mit-b0_torch/nvidia_mit-b0_torch_test.py::MitModuleTest::test_module_static_gpu ``` Error Log (common for cases shown above): ``` E iree.compiler.tools.binaries.CompilerToolError: Error invoking IREE compiler...

- Rename "dynamic" bool field to "shape_type" and provide "static" or "dynamic" as values. - Provide datatype in benchmark results. - Computes and provides comparison with PyTorch (or Tensorflow) for...

Error log for resnet101 that is similar (if not identical) to the error messages produced from the dynamic vulkan case on a few of our PyTorch models: [gist](https://gist.github.com/monorimet/0942bd660e3d7200d34bf6060e552ae2) This error...

Several features/improvements to SHARK's pytest --benchmark option are tracked in this issue: - [x] Improve "frontend" / MLIR dialect argument transmission through SharkBenchmarkRunner - [x] Verify benchmark results for PyTorch+CUDA...

DistilBert fails to lower through torch-mlir pass pipeline (illegal ops)

4

Error output: ``` error: failed to legalize operation 'torch.aten.view' that was explicitly marked illegal note: see current operation: %416 = "torch.aten.view"(%414, %415) : (!torch.vtensor, !torch.list) -> !torch.vtensor Traceback (most recent...

TF tiny-random-flaubert numerics issue on Vulkan. (A100)

1

To reproduce: ``` pytest -s tank/tf/hf_masked_lm/tiny-random-flaubert_tf_test.py -k "vulkan" ``` SHARK results fail to validate against TF golden values: ``` E assert True == False E + where False = compare_tensors_tf(,...

TF roberta/XLM roberta numerics issues on A100 if num_iterations >= 100

2

XLM-roberta assert failure: ``` > np.testing.assert_allclose(golden_out, result, rtol=1e-01, atol=1e-02) E AssertionError: E Not equal to tolerance rtol=0.1, atol=0.01 E E Mismatched elements: 5505 / 4000032 (0.138%) E Max absolute difference:...

In Shark Downloader, we check if local hash for shark_tank artifacts matches upstream hash, and if it doesn't, all artifacts are downloaded from gs://shark_tank for the latest upstream hash, replacing...

`--enable_tf32` is now True by default and certain TF model tests have a switch for relaxing tolerances if TF32 calculations are enabled.