in local development, zip uploads are broken

[inferno-chromium]

|   File "/build/clusterfuzz/src/appengine/handlers/jobs.py", line 132, in post
|     blob_info = self.get_upload()
|   File "/build/clusterfuzz/src/appengine/handlers/base_handler.py", line 317, in get_upload
|     blob_info = storage.GcsBlobInfo.from_key(upload_key)
|   File "/build/clusterfuzz/src/python/google_cloud_utils/storage.py", line 556, in from_key
|     logs.log_error('Failed to get blob from key %s.' % key)
| LogError: Failed to get blob from key 0e3179ad-31b4-4ba5-a45c-86a610e065c7.
| Traceback (most recent call last):
|   File "/build/clusterfuzz/src/python/google_cloud_utils/storage.py", line 554, in from_key
|     return GcsBlobInfo(blobs_bucket(), key)
|   File "/build/clusterfuzz/src/python/google_cloud_utils/storage.py", line 536, in __init__
|     self.filename = gcs_object['metadata'].get(BLOB_FILENAME_METADATA_KEY)
| TypeError: 'NoneType' object is not subscriptable

[oliverchang]

Is the go gcs server running while run_server is running?

[oliverchang]

I just tested this and it worked. If you're accessing the web interface via SSH tunnelling, make sure port 9008 (for the mock GCS server) is forwarded as well.

[inferno-chromium]

I think this was an issue with TEST_BLOBS_BUCKET set for integration tests. This is yet another time this bites me. Removing it works. Any thoughts on fixing this, not a blocker for other people, but creates confusion for us running integration tests

[DelCoding]

in my env, this issue still can reproduce after merged https://github.com/google/clusterfuzz/pull/1726, i using commit id: 1075c6833a26c4b46a28271f63b6878cc5ee4e30.

[inferno-chromium]

@DelCoding - can you try this

$ ps -efww | grep gcs
aarya    155741      1  0 Apr02 ?        00:00:00 /tmp/go-build798608216/b001/exe/gcs -storage-path=storage/local_gcs

Do you see a gcs emulator process ? Do you have golang installed ?

[DelCoding]

yes, output:

❯ ps -efww | grep gcs
junay  8146 10897  0 15:12 pts/1    00:00:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn gcs
junay 18627     1  0 13:14 pts/5    00:00:00 /bin/sh -c go run emulators/gcs.go -storage-path=local/storage
junay 18629 18627  0 13:14 pts/5    00:00:00 go run emulators/gcs.go -storage-path=local/storage
junay 18741 18629  0 13:14 pts/5    00:00:00 /tmp/go-build627719026/b001/exe/gcs -storage-path=local/storage
root     29914  2244  0 14:58 pts/0    00:00:00 /bin/sh -c go run emulators/gcs.go -storage-path=local/storage
root     29916 29914  0 14:58 pts/0    00:00:00 go run emulators/gcs.go -storage-path=local/storage
root     30603 29916  0 14:58 pts/0    00:00:00 /tmp/go-build241825605/command-line-arguments/_obj/exe/gcs -storage-path=local/storage

[inferno-chromium]

Looks like you are running two go processes, can you kill those [one is root and one from junay] and then restart run_server and try again.

[giantbranch]

I have the same problem I am using v2.0.1 git checkout tags/v2.0.1

LogError: Failed to get blob from key fe0e896e-4b85-4385-bc53-13eb0be95772.
| Traceback (most recent call last):
|   File "/root/clusterfuzz/src/python/google_cloud_utils/storage.py", line 557, in from_key
|     return GcsBlobInfo(blobs_bucket(), key)
|   File "/root/clusterfuzz/src/python/google_cloud_utils/storage.py", line 539, in __init__
|     self.filename = gcs_object['metadata'].get(BLOB_FILENAME_METADATA_KEY)
| TypeError: 'NoneType' object is not subscriptable
| ERROR:root:Failed to upload.
| Traceback (most recent call last):
|   File "third_party/webapp2.py", line 604, in dispatch
|     return method(*args, **kwargs)
|   File "/root/clusterfuzz/src/appengine/libs/handler.py", line 287, in wrapper
|     return func(self, *args, **kwargs)
|   File "/root/clusterfuzz/src/appengine/libs/handler.py", line 422, in wrapper
|     return func(self, *args, **kwargs)
|   File "/root/clusterfuzz/src/appengine/handlers/jobs.py", line 132, in post
|     blob_info = self.get_upload()
|   File "/root/clusterfuzz/src/appengine/handlers/base_handler.py", line 319, in get_upload
|     raise helpers.EarlyExitException('Failed to upload.', 500)
| libs.helpers.EarlyExitException: Failed to upload.

gcs is runing.

$ ps -efww | grep gcs
root     11436 11152  0 17:15 pts/13   00:00:00 /bin/sh -c go run emulators/gcs.go -storage-path=/root/clusterfuzz/local/storage/local_gcs
root     11438 11436  0 17:15 pts/13   00:00:00 go run emulators/gcs.go -storage-path=/root/clusterfuzz/local/storage/local_gcs
root     11493 11438  0 17:15 pts/13   00:00:00 /tmp/go-build950734387/b001/exe/gcs -storage-path=/root/clusterfuzz/local/storage/local_gcs
root     11701  3778  0 17:18 pts/1    00:00:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn gcs

[inferno-chromium]

@giantbranch - can you please file a new bug with steps of reproduce, linux distro used, etc (see this issue as an example - https://github.com/google/clusterfuzz/issues/1781). I cannot reproduce this.

[giantbranch]

@inferno-chromium I seem to found the problem by looking at the http request，Because the target of the request to gcs is the localhost，but ClusterFuzz is not running locally.

Local instance of ClusterFuzz can only be accessed locally, not remotely? Or how can I access remotely。

And I found that gcs is also listening locally.....

$ netstat -antp  | grep 9008
tcp        0      0 127.0.0.1:9008          0.0.0.0:*               LISTEN      13881/gcs

[inferno-chromium]

Local setup is not meant for production/remote use. there are some workarounds on https://www.youtube.com/watch?v=65aywI2JcLA feel free to submit a patch if you rely on that support.

[Vancir]

I made the following changes to support uploading files remotely to the local instance of ClusterFuzz.

1. replace the localhost in local/emulators/gcs.go with 0.0.0.0

- http.ListenAndServe(fmt.Sprintf("localhost:%d", port), nil)
+ http.ListenAndServe(fmt.Sprintf("0.0.0.0:%d", port), nil)

Please turn off the firewall rules to make sure you can access the port. You can use the following command to verify whether you can access the port.

$ curl http://<your server ip address>:9008
ERROR: invalid path.

2. relace the localhost in src/local/butler/constants.py with your server ip address

- LOCAL_GCS_SERVER_HOST = "http://localhost:" + str(LOCAL_GCS_SERVER_PORT)
+ LOCAL_GCS_SERVER_HOST = "http://" + "<your server ip address here>" + ":" + str(LOCAL_GCS_SERVER_PORT)

Of course, you can run the following python code to get your server ip address automatically.

import socket 
address = socket.gethostbyname(socket.gethostname())
print(address)