Verbose Isn't Verbose Enough To Help

[rgajason]

GitGuardian Shield Version

• [X] I can reproduce this bug in the latest version

Command executed

ggshield --verbose scan --show-secrets --verbose repo [email protected]:myorg/myrepo.git

Describe the bug

--verbose flag shows the full stack trace, but doesn't indicate what specific iteration was being performed when the error was thrown.

Expected behavior

Enough data provided to narrow down what specific data was causing the issue (such as, which file/endpoint was being hit).

Traceback (if available)

In the below, I need to troubleshoot both the timeout and the "this field may not be blank" error, but I don't have enough info to narrow down either.

% ggshield --verbose scan --show-secrets --verbose repo [email protected]:myorg/myrepo.git
Scanning Commits  [------------------------------------]    2%  00:01:36
Error scanning. Results may be incomplete.
Add the following files to your paths-ignore:
- : filename:: [ErrorDetail(string='This field may not be blank.', code='blank')]
Scanning Commits  [#####-------------------------------]   15%  00:03:53
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1079, in _send_output
    self.send(chunk)
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1001, in send
    self.sock.sendall(data)
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1173, in send
    return self._sslobj.write(data)
socket.timeout: The write operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.9/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1079, in _send_output
    self.send(chunk)
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1001, in send
    self.sock.sendall(data)
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1173, in send
    return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', timeout('The write operation timed out'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/ggshield/dev_scan.py", line 46, in scan_repo_path
    return scan_commit_range(
  File "/usr/local/lib/python3.9/site-packages/ggshield/dev_scan.py", line 261, in scan_commit_range
    scans.append(future.result())
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/site-packages/ggshield/dev_scan.py", line 196, in scan_commit
    results = commit.scan(
  File "/usr/local/lib/python3.9/site-packages/ggshield/scan/scannable.py", line 146, in scan
    scan = future.result()
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/site-packages/pygitguardian/client.py", line 295, in multi_content_scan
    resp = self.post(
  File "/usr/local/lib/python3.9/site-packages/pygitguardian/client.py", line 205, in post
    return self.request(
  File "/usr/local/lib/python3.9/site-packages/pygitguardian/client.py", line 141, in request
    response: Response = self.session.request(
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', timeout('The write operation timed out'))
Error: ('Connection aborted.', timeout('The write operation timed out'))

[agateau-gg]

Looks like there are two issues here:

1. When scanning with --verbose, ggshield should print the name of the files being scanned
2. In case of a network error, ggshield should print an informative message (with the name of the file being scanned) instead of dumping the call stack of the failed network request

Does this make sense?

[rgajason]

That sounds reasonable to me. I would tweak point 2 a bit to say "with the network endpoint" (I don't know if it is timing out talking to GitHub or the GG API), but I have a feeling that is what you meant and I'm just being nit picky...

[agateau-gg]

scan repo clones the repository first, then scans it, so if it fails in the middle of a scan you can be sure it's a timing out while talking to GitGuardian API.

I am going to create separate issues for the progress while scanning and for error reporting, and link them to this issue.

[agateau-gg]

Finally closing this one now that #212 is done.