fuzzbench
fuzzbench copied to clipboard
Published
20 hours ago •
google
google
Find out why logging doesn't work on service instance.
It fails with this:
Traceback (most recent call last):
File "/tmp/fuzzbench-automatic-experiment-repo/common/retry.py", line 81, in _wrapper
result = func(*args, **kwargs)
File "/tmp/fuzzbench-automatic-experiment-repo/common/logs.py", line 191, in _report_error_with_retries
_error_reporting_client.report(message)
File "/tmp/fuzzbench-automatic-experiment-repo/.venv/lib/python3.8/site-packages/google/cloud/error_reporting/client.py", line 357, in report
self._send_error_report(
File "/tmp/fuzzbench-automatic-experiment-repo/.venv/lib/python3.8/site-packages/google/cloud/error_reporting/client.py", line 317, in _send_error_report
self.report_errors_api.report_error_event(error_report)
File "/tmp/fuzzbench-automatic-experiment-repo/.venv/lib/python3.8/site-packages/google/cloud/error_reporting/_gapic.py", line 73, in report_error_event
self._gapic_api.report_error_event(
File "/tmp/fuzzbench-automatic-experiment-repo/.venv/lib/python3.8/site-packages/google/cloud/errorreporting_v1beta1/services/report_errors_service/client.py", line 432, in report_error_event
response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
File "/tmp/fuzzbench-automatic-experiment-repo/.venv/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
return wrapped_func(*args, **kwargs)
File "/tmp/fuzzbench-automatic-experiment-repo/.venv/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 69, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.PermissionDenied: 403 Request had insufficient authentication scopes.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/fuzzbench-automatic-experiment-repo/.venv/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
return callable_(*args, **kwargs)
File "/tmp/fuzzbench-automatic-experiment-repo/.venv/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/tmp/fuzzbench-automatic-experiment-repo/.venv/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.PERMISSION_DENIED
details = "Request had insufficient authentication scopes."
debug_error_string = "{"created":"@1637242902.586894691","description":"Error received from peer ipv4:74.125.132.95:443","file":"src/core/lib/surface/call.cc","file_line":1063,"grpc_message":"Request had insufficient authentication scopes.","grpc_status":7}"
Also, we need to make sure if retry is called from logs.py that it does not log.
I worked around this by setting FORCE_LOCAL=True in the environment when calling the script so that logs are sent to stderr instead of google cloud
I am trying to run fuzzbench local experiment on GCP VM instance but sometimes it just get stuck in dispatcher and didn't launch experiment. I have to interrupt the process with ctrl+c and see Python throws similar exception:
third_party/sancov.py
^CTraceback (most recent call last):
File "/work/src/experiment/dispatcher.py", line 212, in <module>
sys.exit(main())
File "/work/src/experiment/dispatcher.py", line 194, in main
dispatcher_main()
File "/work/src/experiment/dispatcher.py", line 148, in dispatcher_main
trials = build_images_for_trials(experiment.fuzzers, experiment.benchmarks,
File "/work/src/experiment/dispatcher.py", line 118, in build_images_for_trials
build_successes = builder.build_all_fuzzer_benchmarks(fuzzers, benchmarks)
File "/work/src/experiment/build/builder.py", line 192, in build_all_fuzzer_benchmarks
successful_calls = retry_build_loop(build_fuzzer_benchmark,
File "/work/src/experiment/build/builder.py", line 148, in retry_build_loop
results = pool.starmap(build_func, inputs)
File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 375, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 768, in get
self.wait(timeout)
File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 765, in wait
self._event.wait(timeout)
File "/usr/local/lib/python3.10/threading.py", line 607, in wait
signaled = self._cond.wait(timeout)
File "/usr/local/lib/python3.10/threading.py", line 320, in wait
waiter.acquire()
KeyboardInterrupt
Waiting up to 5 seconds.
Sent all pending logs.
^CTraceback (most recent call last):
File "/home/ubuntu/fuzzbench/.venv/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 65, in error_remapped_callable
return callable_(*args, **kwargs)
File "/home/ubuntu/fuzzbench/.venv/lib/python3.10/site-packages/grpc/_channel.py", line 1161, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/home/ubuntu/fuzzbench/.venv/lib/python3.10/site-packages/grpc/_channel.py", line 1004, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.PERMISSION_DENIED
details = "Request had insufficient authentication scopes."
debug_error_string = "UNKNOWN:Error received from peer ipv4:74.125.126.95:443 {grpc_message:"Request had insufficient authentication scopes.", grpc_status:7, c
reated_time:"2023-11-10T04:49:08.053930896+00:00"}"
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ubuntu/fuzzbench/common/retry.py", line 81, in _wrapper
result = func(*args, **kwargs)
File "/home/ubuntu/fuzzbench/common/logs.py", line 196, in _report_error_with_retries
_error_reporting_client.report(message)
File "/home/ubuntu/fuzzbench/.venv/lib/python3.10/site-packages/google/cloud/error_reporting/client.py", line 355, in report
self._send_error_report(
File "/home/ubuntu/fuzzbench/.venv/lib/python3.10/site-packages/google/cloud/error_reporting/client.py", line 315, in _send_error_report
self.report_errors_api.report_error_event(error_report)
File "/home/ubuntu/fuzzbench/.venv/lib/python3.10/site-packages/google/cloud/error_reporting/_gapic.py", line 75, in report_error_event
self._gapic_api.report_error_event(
File "/home/ubuntu/fuzzbench/.venv/lib/python3.10/site-packages/google/cloud/errorreporting_v1beta1/services/report_errors_service/client.py", line 535, in report_erro
r_event
response = rpc(
File "/home/ubuntu/fuzzbench/.venv/lib/python3.10/site-packages/google/api_core/gapic_v1/method.py", line 113, in __call__
return wrapped_func(*args, **kwargs)
File "/home/ubuntu/fuzzbench/.venv/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.PermissionDenied: 403 Request had insufficient authentication scopes. [reason: "ACCESS_TOKEN_SCOPE_INSUFFICIENT"
domain: "googleapis.com"
metadata {
key: "method"
value: "google.devtools.clouderrorreporting.v1beta1.ReportErrorsService.ReportErrorEvent"
}
metadata {
key: "service"
value: "clouderrorreporting.googleapis.com"
}
]
Following this thread - I have exactly same issue with local experiment on gcloud instance
Did you figure out how to move pass sancov.py in your experiment? @chinggg I'm facing the same issue and would appreciate any advice!