[go-example] object doesn't exist

[ncarenton]

Thanks for this project!

I can not make the go-example work, it finds the bucket but not the object:

docker run -d --name fake-gcs-server -p 4443:4443 -v ${PWD}/examples/data:/data fsouza/fake-gcs-server
763d400668bf735e067fc61b0983a49e4393eb673ae9473379e961faf050a90b
cd examples/go
go build
./go-example 
buckets: [sample-bucket]
2020/06/19 19:31:47 storage: object doesn't exist

The strange thing is that curl is working fine so the object is actually there:

 curl --insecure https://127.0.0.1:4443/storage/v1/b/sample-bucket/o
{"kind":"storage#objects","items":[{"kind":"storage#object","name":"some_file.txt","id":"sample-bucket/some_file.txt","bucket":"sample-bucket","size":"33","timeCreated":"2020-06-19T17:41:33Z","timeDeleted":"0001-01-01T00:00:00Z","updated":"2020-06-19T17:41:33Z","generation":"0"}]}

It is also working fine with the python example:

 python python.py 
Bucket: sample-bucket

Blob: some_file.txt
b'Some amazing content to be loaded' 

[fsouza]

Hey @ncarenton, thanks for reporting and apologies for the delayed response. Not sure if you noticed this comment in the example file: https://github.com/fsouza/fake-gcs-server/blob/c822c3b926f90755b38471be4b77d422d3623007/examples/go/main.go#L5-L9

(I feel like we can probably move that comment elsewhere in the file, it's easy to miss it as part of the package statement)

In order for the downloads to work in the way the Go example expects, the server needs to be started with some flags.

[ncarenton]

Hi @fsouza, no worries for the short delay :)

I actually did miss this comment, it could indeed be moved to downloadFile() for example, for better visibility.

Starting my docker container with -public-host does fix my issue! Still I would like to ask if it is possible to use a regular GCS client with the fake server, meaning without these 2 arguments: option.WithEndpoint("https://storage.gcs.127.0.0.1.nip.io:4443/storage/v1/"), option.WithHTTPClient(httpClient) ? Is it possible to do that and set STORAGE_EMULATOR_HOST env var so we start the fake server without -public-host then ? I get the feeling it would be simpler.

Many thanks for your help

[fsouza]

I don't know if the Go sdk supports STORAGE_EMULATOR_HOST 🤔 Does it?

If the sdk supports that env var, we could support it in the emulator too, or figure out a way to bind to any host (not sure if it's possible because of conflicting paths).

[ncarenton]

I guess so: https://github.com/googleapis/google-cloud-go/blob/master/storage/storage.go#L104

[fsouza]

Nice, in that case we need to look for that variable and if it's defined, bind in the proper port and use HTTP instead of HTTPs.

[yiyue115]

Hello 😃 Actually I faced a similar issue as well cuz I missed that comment 😛 Is there any update on this one?

[OrangeFlag]

Hello, in my case I have set the correct external-url and public-host for my docker container, but when I call fake-gcs from the local machine, I also get a 404

I used wireshark and realized that the requests differ in the Host field and response in Location field. Wouldn't it be nice to reply based on the Host being sent and remove the external-url and public-host? So it would be possible to call the service from any environment

Thanks for your work!

TLDR

docker-compose for fast-gcs

gcs:
    image: fsouza/fake-gcs-server
    command: -backend memory -scheme http -port 4443 -public-host gcs:4443 -external-url http://gcs:4443
    ports:
      - 4443:4443

Fast fix to request fake-gcs from localhost:

storage.NewClient(ctx,
		option.WithEndpoint("http://localhost:4443/storage/v1/"),
		option.WithoutAuthentication(),
		option.WithHTTPClient(&http.Client{
			Transport: &HostFixRoundTripper{&http.Transport{}},
		}))
//...
//...
type HostFixRoundTripper struct {
	Proxy http.RoundTripper
}

func (l HostFixRoundTripper) RoundTrip(request *http.Request) (*http.Response, error) {
	request.Host = "gcs:4443"
	res, err := l.Proxy.RoundTrip(request)
	if res != nil {
		location := res.Header.Get("Location")
		if len(location) != 0 {
			res.Header.Set("Location", strings.Replace(location, "gcs", "localhost", 1))
		}
	}
	return res, err
}

[sfro]

I'm not able to create objects no matter what public-host I put 🤔 logs show that the request isn't even reaching the server.

Using cloud.google.com/go/storage v1.14.0.

[fsouza]

Hey @sfro, I'll try updating the example to include upload. thanks for reporting!