gcp-storage-emulator icon indicating copy to clipboard operation
gcp-storage-emulator copied to clipboard

Bucket or Object not found

Open Raffaello opened this issue 2 years ago • 4 comments

Describe the bug when do a download of an object is reported 404, not found.

To Reproduce

  • run the docker container as a per readme example
  • create in the shared foder, cloudstorage, my-bucket/my-filename
  • use a snippet program to download it and it won't be found even something like:
import os



from google.cloud import exceptions, storage



HOST = "localhost"

PORT = 9023

BUCKET = "my-bucket"



os.environ["STORAGE_EMULATOR_HOST"] = f"http://{HOST}:{PORT}"

client = storage.Client()



#try:

#    bucket = client.create_bucket(BUCKET)

#except exceptions.Conflict:

#    bucket = client.bucket(BUCKET)



bucket = client.bucket(BUCKET)

blob = bucket.blob("my-filename")

#blob.upload_from_string("test1")

print(blob.download_as_bytes())


Expected behavior A clear and concise description of what you expected to happen.

If the file is present in the docker folder /storage/my-bucket/my-filename it should be expected to be found and downloaded.

System (please complete the following information):

  • OS version:
  • Python version:
  • gcp-storage-emulator version:

Additional context it looks like the file must be uploaded from calling the api, cannot be manually added in the cloud storage folder. Anyway this is just a basic use case, having a file to be downloaded, it shouldn't require to be uploaded first.

Raffaello avatar May 03 '22 14:05 Raffaello

If you just manually place a file, the metadata won't exists. That's why it can't be found.

oittaa avatar May 03 '22 20:05 oittaa

If you just manually place a file, the metadata won't exists. That's why it can't be found.

wouldn't be possible to use libevent or similar to monitor the /storage folder and based on a file creation or deletion or whatever, auto generate the metadata?

Raffaello avatar May 03 '22 22:05 Raffaello

If you just manually place a file, the metadata won't exists. That's why it can't be found.

wouldn't be possible to use libevent or similar to monitor the /storage folder and based on a file creation or deletion or whatever, auto generate the metadata?

anyway it could be achieved with a side utility too as a pre-step, instead of copy the file do an upload.

Raffaello avatar May 03 '22 22:05 Raffaello

If you just manually place a file, the metadata won't exists. That's why it can't be found.

wouldn't be possible to use libevent or similar to monitor the /storage folder and based on a file creation or deletion or whatever, auto generate the metadata?

anyway it could be achieved with a side utility too as a pre-step, instead of copy the file do an upload.

the only concerns is when using a volume that has some predefined files for example and then starting the emulator would be pointless as there won't be any metadata on the content of the volume, so a scanning of the volumne to creates buckkets from folders and objects from file could be useful.

Raffaello avatar May 03 '22 22:05 Raffaello