Add bloom test

[AmosLewis]

[monorimet]

Let's merge this and run the nightly so the checks go green if it's pushed to a branch of the main repo we can run the nightly on it during PR and have the new model in the cloud but not if it's forked

[AmosLewis]

➜  SHARK git:(bloom-test) ✗ pytest tank/test_models.py                            
======================================= test session starts =======================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0 -- /home/chi/src/ubuntu20/shark/SHARK/shark.venv/bin/python3
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/chi/src/ubuntu20/shark/SHARK/.hypothesis/examples')
rootdir: /home/chi/src/ubuntu20/shark/SHARK, configfile: pytest.ini
plugins: xdist-2.5.0, forked-1.4.0, hypothesis-6.54.6
collected 4 items                                                                                 

tank/test_models.py::SharkModuleTest::test_module_bigscience_bloom_560m_torch_dynamic_cpu SKIPPED [ 25%]
tank/test_models.py::SharkModuleTest::test_module_bigscience_bloom_560m_torch_dynamic_cuda SKIPPED [ 50%]
tank/test_models.py::SharkModuleTest::test_module_bigscience_bloom_560m_torch_static_cpu PASSED [ 75%]
tank/test_models.py::SharkModuleTest::test_module_bigscience_bloom_560m_torch_static_cuda PASSED [100%]

============================ 2 passed, 2 skipped in 222.27s (0:03:42) =============================

[AmosLewis]

pytest tank/test_models.py -s --benchmark will got this bug. Looks like we need to add "tm_tensor " support for benchmark? @monorimet @dan-garvey

pytest tank/test_models.py -s --benchmark
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tank/test_models.py:357: in test_module
    self.module_tester.create_and_check_module(dynamic, device)
tank/test_models.py:187: in create_and_check_module
    self.benchmark_module(shark_module, inputs, dynamic, device)
tank/test_models.py:199: in benchmark_module
    shark_module.shark_runner.benchmark_all_csv(
shark/shark_benchmark_runner.py:270: in benchmark_all_csv
    self.setup_cl(inputs)
shark/shark_benchmark_runner.py:68: in setup_cl
    self.benchmark_cl = build_benchmark_args(
shark/iree_utils/benchmark_utils.py:73: in build_benchmark_args
    mlir_input_types = tensor_to_type_str(input_tensors, mlir_dialect)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

input_tensors = (array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1,
        0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1,...0, 1, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0,
        0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1]]),)
mlir_dialect = 'tm_tensor'

    def tensor_to_type_str(input_tensors: tuple, mlir_dialect: str):
        """
        Input: A tuple of input tensors i.e tuple(torch.tensor)
        Output: list of string that represent mlir types (i.e 1x24xf64)
        # TODO: Support more than floats, and ints
        """
        list_of_type = []
        for input_tensor in input_tensors:
            type_string = "x".join([str(dim) for dim in input_tensor.shape])
            if mlir_dialect in ["linalg", "tosa"]:
                dtype_string = str(input_tensor.dtype).replace("torch.", "")
            elif mlir_dialect in ["mhlo", "tflite"]:
                dtype = input_tensor.dtype
                try:
                    dtype_string = re.findall("'[^\"]*'", str(dtype))[0].replace(
                        "'", ""
                    )
                except IndexError:
                    dtype_string = str(dtype)
            regex_split = re.compile("([a-zA-Z]+)([0-9]+)")
>           match = regex_split.match(dtype_string)
E           UnboundLocalError: local variable 'dtype_string' referenced before assignment

[monorimet]

Yes, we need to add support for this dialect.

[AmosLewis]

For now, we just skip the bloom benchmark by

        if config["model_name"] == "bigscience/bloom-560m" and self.module_tester.benchmark  == True:
            pytest.skip(reason="tm_tensor dialect in bloom not supported for benchmark.")

But this bug should be fixed later in https://github.com/nod-ai/SHARK/issues/379

[AmosLewis]

The vulkan is also not supported. Right now, we mark it xfail. https://github.com/nod-ai/SHARK/issues/380

        if config["model_name"] == "bigscience/bloom-560m" and device == "vulkan":
            pytest.xfail(
                reason="vulkan not supported with tm_tensor in bloom, https://github.com/nod-ai/SHARK/issues/380"
            )