Locally generated shark_tank artifacts are not usable for pytests.
In Shark Downloader, we check if local hash for shark_tank artifacts matches upstream hash, and if it doesn't, all artifacts are downloaded from gs://shark_tank for the latest upstream hash, replacing local files. This becomes a problem if one uses generate_sharktank.py to populate local shark_tank and run tests, as the upstream artifacts are used instead of the local artifacts (in my case, with significant changes).
I think this is of critical importance to our correspondence with the IREE team as well as our SHARK team's development process. We have a few options to handle this:
- add a pytest option to use local files (avoid SHARK downloader)
- have shark downloader look in SHARK/gen_shark_tank/ before doing anything with google storage -- if locally generated artifacts are present, don't touch gs://shark_tank and simply use local artifacts.
to reproduce:
python generate_sharktank.py
pytest -s tank/MiniLM-L12-H384-uncased/
it will be evident that the artifacts are replaced by contents of gs://shark_tank/microsoft_MiniLM-L12-H384-uncased_tf/