Some old artifacts are unreachable

[Janus-Shiau]

Recently I found some of my previous logged artifacts are unreachable.

They are still on the UI, but I can't download these files.

If I using wandb api to download, for example:

import wandb
run = wandb.init()
artifact = run.use_artifact('advance-nn/pose/from_scratch:v0', type='model')
artifact_dir = artifact.download()

I'll some error as follows:

wandb: Downloading large artifact from_scratch:v0, 558.40MB. 15 files... Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jiayau/anaconda3/envs/tf-2.7/lib/python3.8/site-packages/wandb/apis/public.py", line 4381, in download
    pool.map(partial(self._download_file, root=dirpath), manifest.entries)
  File "/home/jiayau/anaconda3/envs/tf-2.7/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/jiayau/anaconda3/envs/tf-2.7/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/home/jiayau/anaconda3/envs/tf-2.7/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/jiayau/anaconda3/envs/tf-2.7/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/jiayau/anaconda3/envs/tf-2.7/lib/python3.8/site-packages/wandb/apis/public.py", line 4475, in _download_file
    return self.get_path(name).download(root)
  File "/home/jiayau/anaconda3/envs/tf-2.7/lib/python3.8/site-packages/wandb/apis/public.py", line 3847, in download
    cache_path = manifest.storage_policy.load_file(
  File "/home/jiayau/anaconda3/envs/tf-2.7/lib/python3.8/site-packages/wandb/sdk/wandb_artifacts.py", line 909, in load_file
    response.raise_for_status()
  File "/home/jiayau/anaconda3/envs/tf-2.7/lib/python3.8/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://10.112.34.105:8080/local-files/wandb_artifacts/41/437/dc44f5dcf434e80f5634212d31ab97d9?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=lsABaMIF90rd0d8kQZAacQ%3D%3D%2F20220805%2Fwandb-local%2Fs3%2Faws4_request&X-Amz-Date=20220805T030928Z&X-Amz-Expires=60&X-Amz-SignedHeaders=host&X-User=advance-nn&response-content-disposition=attachment&X-Amz-Signature=31d07e6cf757ecbd618ea2d2a46b24632b7463baab17892d90bc8e1f684b1cc7

My local instance is running on Ubuntu 16.04, and its version is 0.15.0. My client side is running on Ubuntu 16.04 or 18.04, and its version is 0.12.21 and 0.13.0.

I also check the file system, assuming these artifacts are stored at minio/local-files/wandb_artifacts. There is actually no such a folder called 41/437.

I hope this issue can be reproduced and solved soon.

[vanpelt]

This means you're volume either changed or was dropped. By default wandb server mounts a docker volume named "wandb" into the container that persists all data. If the machine itself you were running on changed, or that volume was removed all the data within would be lost.