google
Production setup clusterfuzz - issue with accessing metadata service
I am seeing the below error after setting up clusterfuzz when visiting the home page:
Maximum number of 3 retries exceeded while calling <function make_call.<locals>.rpc_call at 0x3e1aab2c23b0>, last exception: 503 Getting metadata from plugin failed with error: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdatastore from the Google Compute Engine metadata service. Status: 500 Response:\nb'Could not fetch URI /computeMetadata/v1/instance/service-accounts/[email protected]/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdatastore\\n'", <google.auth.transport.requests._Response object at 0x3e1aa73d96d0>) Traceback (most recent call last): File "/srv/handlers/base_handler.py", line 273, in dispatch_request return super(Handler, self).dispatch_request(*args, **kwargs) File "/layers/google.python.pip/pip/lib/python3.7/site-packages/flask/views.py", line 163, in dispatch_request return meth(*args, **kwargs) File "/srv/libs/handler.py", line 429, in wrapper response = make_response(func(self, *args, **kwargs)) File "/srv/handlers/testcase_list.py", line 185, in get result, params = get_result() File "/srv/handlers/testcase_list.py", line 120, in get_result 'fuzzer_name_indices') File "/srv/libs/crash_access.py", line 85, in add_scope scope = get_scope() File "/srv/libs/crash_access.py", line 58, in get_scope allowed_job_type = access.get_user_job_type() File "/srv/libs/access.py", line 53, in get_user_job_type privileged_user_emails = (db_config.get_value('privileged_users') or File "/srv/clusterfuzz/_internal/config/db_config.py", line 36, in get_value config = get() File "/srv/clusterfuzz/_internal/config/db_config.py", line 31, in get return data_types.Config.query().get() File "third_party/google/cloud/ndb/query.py", line 1214, in wrapper return wrapped(self, *dummy_args, _options=query_options) File "third_party/google/cloud/ndb/utils.py", line 121, in wrapper return wrapped(*args, **new_kwargs) File "third_party/google/cloud/ndb/utils.py", line 153, in positional_wrapper return wrapped(*args, **kwds) File "third_party/google/cloud/ndb/query.py", line 2077, in get return self.get_async(_options=kwargs["_options"]).result() File "third_party/google/cloud/ndb/tasklets.py", line 214, in result self.check_success() File "third_party/google/cloud/ndb/tasklets.py", line 161, in check_success raise self._exception File "third_party/google/cloud/ndb/tasklets.py", line 334, in _advance_tasklet yielded = self.generator.throw(type(error), error, traceback) File "third_party/google/cloud/ndb/query.py", line 2111, in get_async results = yield _datastore_query.fetch(options) File "third_party/google/cloud/ndb/tasklets.py", line 334, in _advance_tasklet yielded = self.generator.throw(type(error), error, traceback) File "third_party/google/cloud/ndb/_datastore_query.py", line 113, in fetch while (yield results.has_next_async()): File "third_party/google/cloud/ndb/tasklets.py", line 334, in _advance_tasklet yielded = self.generator.throw(type(error), error, traceback) File "third_party/google/cloud/ndb/_datastore_query.py", line 340, in has_next_async yield self._next_batch() # First time File "third_party/google/cloud/ndb/tasklets.py", line 334, in _advance_tasklet yielded = self.generator.throw(type(error), error, traceback) File "third_party/google/cloud/ndb/_datastore_query.py", line 370, in _next_batch response = yield _datastore_run_query(query) File "third_party/google/cloud/ndb/tasklets.py", line 334, in _advance_tasklet yielded = self.generator.throw(type(error), error, traceback) File "third_party/google/cloud/ndb/_datastore_query.py", line 1019, in _datastore_run_query "RunQuery", request, timeout=query.timeout File "third_party/google/cloud/ndb/tasklets.py", line 338, in _advance_tasklet yielded = self.generator.send(send_value) File "third_party/google/cloud/ndb/_retry.py", line 111, in retry_wrapper cause=error, google.api_core.exceptions.RetryError: Maximum number of 3 retries exceeded while calling <function make_call.<locals>.rpc_call at 0x3e1aab2c23b0>, last exception: 503 Getting metadata from plugin failed with error: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdatastore from the Google Compute Engine metadata service. Status: 500 Response:\nb'Could not fetch URI /computeMetadata/v1/instance/service-accounts/[email protected]/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdatastore\\n'", <google.auth.transport.requests._Response object at 0x3e1aa73d96d0>)
I tried also clicking other pages like fuzzer statistics and login. After successful login, the page looks empty and the network traffic shows the request to /session-login has failed with 500. Below is the response, which is similar to above:
{ "message": "(\"Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdatastore%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Ffirebase%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fidentitytoolkit%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email from the Google Compute Enginemetadata service. Status: 500 Response:\\nb'Could not fetch URI /computeMetadata/v1/instance/service-accounts/[email protected]/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdatastore%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Ffirebase%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fidentitytoolkit%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email\\\\n'\", <google.auth.transport.requests._Response object at 0x3ec8e0701850>)", "email": "", "traceDump": "Traceback (most recent call last):\n File \"/layers/google.python.pip/pip/lib/python3.7/site-packages/google/auth/compute_engine/credentials.py\", line 113, in refresh\n request, service_account=self._service_account_email, scopes=scopes\n File \"/layers/google.python.pip/pip/lib/python3.7/site-packages/google/auth/compute_engine/_metadata.py\", line 263, in get_service_account_token\n token_json = get(request, path, params=params)\n File \"/layers/google.python.pip/pip/lib/python3.7/site-packages/google/auth/compute_engine/_metadata.py\", line 187, in get\n response,\ngoogle.auth.exceptions.TransportError: (\"Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdatastore%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Ffirebase%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fidentitytoolkit%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email from the Google Compute Enginemetadata service. Status: 500 Response:\\nb'Could not fetch URI /computeMetadata/v1/instance/service-accounts/[email protected]/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdatastore%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Ffirebase%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fidentitytoolkit%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email\\\\n'\", <google.auth.transport.requests._Response object at 0x3ec8e0701850>)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/srv/handlers/base_handler.py\", line 273, in dispatch_request\n return super(Handler, self).dispatch_request(*args, **kwargs)\n File \"/layers/google.python.pip/pip/lib/python3.7/site-packages/flask/views.py\", line 163, in dispatch_request\n return meth(*args, **kwargs)\n File \"/srv/libs/handler.py\", line 399, in wrapper\n response = make_response(func(self))\n File \"/srv/handlers/login.py\", line 58, in post\n session_cookie = auth.create_session_cookie(id_token, expires_in)\n File \"/srv/libs/auth.py\", line 173, in create_session_cookie\n return auth.create_session_cookie(id_token, expires_in=expires_in)\n File \"/layers/google.python.pip/pip/lib/python3.7/site-packages/firebase_admin/auth.py\", line 173, in create_session_cookie\n return token_generator.create_session_cookie(id_token, expires_in)\n File \"/layers/google.python.pip/pip/lib/python3.7/site-packages/firebase_admin/_token_gen.py\", line 209, in create_session_cookie\n response = self.client.body('post', ':createSessionCookie', json=payload)\n File \"/layers/google.python.pip/pip/lib/python3.7/site-packages/firebase_admin/_http_client.py\", line 113, in body\n resp = self.request(method, url, **kwargs)\n File \"/layers/google.python.pip/pip/lib/python3.7/site-packages/firebase_admin/_http_client.py\", line 104, in request\n resp = self._session.request(method, self._base_url + url, **kwargs)\n File \"/layers/google.python.pip/pip/lib/python3.7/site-packages/google/auth/transport/requests.py\", line 476, in request\n self.credentials.before_request(auth_request, method, url, request_headers)\n File \"/layers/google.python.pip/pip/lib/python3.7/site-packages/google/auth/credentials.py\", line 133, in before_request\n self.refresh(request)\n File \"/layers/google.python.pip/pip/lib/python3.7/site-packages/google/auth/compute_engine/credentials.py\", line 117, in refresh\n six.raise_from(new_exc, caught_exc)\n File \"<string>\", line 3, in raise_from\ngoogle.auth.exceptions.RefreshError: (\"Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdatastore%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Ffirebase%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fidentitytoolkit%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email from the Google Compute Enginemetadata service. Status: 500 Response:\\nb'Could not fetch URI /computeMetadata/v1/instance/service-accounts/[email protected]/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdatastore%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.read_write%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Ffirebase%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fidentitytoolkit%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email\\\\n'\", <google.auth.transport.requests._Response object at 0x3ec8e0701850>)\n", "status": 500, "type": "RefreshError" }
Any insights in troubleshooting this would be helpful. I have tried editing permissions of firebase service account to provide more privileges to no avail.
Hi @dbdante, this seems to indicate that the frontend is getting confused and thinking it's running on Compute Engine instead of App Engine. Was this deployed to App Engine?
Thanks for the response @oliverchang. I am using App Engine.
I found a work around. Although the console was showing that firebase-adminsdk email is the service account for "default" and "cron service", the app engine seemed to use the default app engine service account. I was able to fix the issue by adding the following:
service_account: [email protected]
to app.yaml and cron-service.yaml.
This seems to be appengine behavior as I was able to reproduce it with a simple app by deploying it without the service account line. Once the above line was added, metaservice started functioning right.