SHARK icon indicating copy to clipboard operation
SHARK copied to clipboard

Locally generated shark_tank artifacts are not usable for pytests.

Open monorimet opened this issue 3 years ago • 0 comments

In Shark Downloader, we check if local hash for shark_tank artifacts matches upstream hash, and if it doesn't, all artifacts are downloaded from gs://shark_tank for the latest upstream hash, replacing local files. This becomes a problem if one uses generate_sharktank.py to populate local shark_tank and run tests, as the upstream artifacts are used instead of the local artifacts (in my case, with significant changes).

I think this is of critical importance to our correspondence with the IREE team as well as our SHARK team's development process. We have a few options to handle this:

  1. add a pytest option to use local files (avoid SHARK downloader)
  2. have shark downloader look in SHARK/gen_shark_tank/ before doing anything with google storage -- if locally generated artifacts are present, don't touch gs://shark_tank and simply use local artifacts.

to reproduce:

python generate_sharktank.py
pytest -s tank/MiniLM-L12-H384-uncased/

it will be evident that the artifacts are replaced by contents of gs://shark_tank/microsoft_MiniLM-L12-H384-uncased_tf/

monorimet avatar Aug 15 '22 19:08 monorimet