[CLI]: wandb: Network error (SSLError), entering retry loop.

[yc-cui]

Describe the bug

weird problem, just

wandb.init()

and I got

Retry attempt failed:
Traceback (most recent call last):
  File "/usr/local/python3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen
    chunked=chunked,
  File "/usr/local/python3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/usr/local/python3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
    conn.connect()
  File "/usr/local/python3/lib/python3.7/site-packages/urllib3/connection.py", line 424, in connect
    tls_in_tls=tls_in_tls,
  File "/usr/local/python3/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 450, in ssl_wrap_socket
    sock, context, tls_in_tls, server_hostname=server_hostname
  File "/usr/local/python3/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/python3/lib/python3.7/ssl.py", line 423, in wrap_socket
    session=session
  File "/usr/local/python3/lib/python3.7/ssl.py", line 870, in _create
    self.do_handshake()
  File "/usr/local/python3/lib/python3.7/ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate is not yet valid (_ssl.c:1076)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/python3/lib/python3.7/site-packages/requests/adapters.py", line 450, in send
    timeout=timeout
  File "/usr/local/python3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 786, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/python3/lib/python3.7/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.wandb.ai', port=443): Max retries exceeded with url: /graphql (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate is not yet valid (_ssl.c:1076)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/python3/lib/python3.7/site-packages/wandb/sdk/lib/retry.py", line 102, in __call__
    result = self._call_fn(*args, **kwargs)
  File "/usr/local/python3/lib/python3.7/site-packages/wandb/sdk/internal/internal_api.py", line 140, in execute
    return self.client.execute(*args, **kwargs)
  File "/usr/local/python3/lib/python3.7/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 52, in execute
    result = self._get_result(document, *args, **kwargs)
  File "/usr/local/python3/lib/python3.7/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/client.py", line 60, in _get_result
    return self.transport.execute(document, *args, **kwargs)
  File "/usr/local/python3/lib/python3.7/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/transport/requests.py", line 38, in execute
    request = requests.post(self.url, **post_args)
  File "/usr/local/python3/lib/python3.7/site-packages/requests/api.py", line 117, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/local/python3/lib/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/python3/lib/python3.7/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/python3/lib/python3.7/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/python3/lib/python3.7/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='api.wandb.ai', port=443): Max retries exceeded with url: /graphql (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate is not yet valid (_ssl.c:1076)')))
wandb: Network error (SSLError), entering retry loop.

However, yesterday the code is running approciately. I didn't use any proxy in my computer.

Additional Files

No response

Environment

WandB version: 0.12.16

OS: Centos 7.6

Python version: 3.7.6

Additional Context

No response

[nate-wandb]

Hi @cuiyc2000, is it possible that there were any changes to your local network that could have causes this? It seems that something may be terminating SSL before it reaches your machine. Are you able to try this on another network such as a mobile hotspot and see if you encounter the same issue?

I'm able to create runs to api.wandb.ai without any issues right now so this doesn't seem to be an issue with our certification manager.

[vanpelt]

The error looks like clock skew. Can you be sure the time is correct on whatever server you're connecting from?

[yc-cui]

sorry for my mistake. Indeed it's clock skew. @vanpelt @nate-wandb

[jjahanip]

The issue is because of the network's network's firewall SSL decryption [ref]. To solve the problem, you can try the following:

Windows: pip install pip-system-certs Linux: add os.environ['SSL_CERT_DIR'] = '/etc/ssl/certs' and os.environ['REQUESTS_CA_BUNDLE'] = '/etc/ssl/certs/ca-certificates.crt' to your code before wandb.init()

[fa00fa]

hi!, @cuiyc2000 ,How can I solve this clock skew problem, I have the same problem as you?

[yc-cui]

hi!, @cuiyc2000 ,How can I solve this clock skew problem, I have the same problem as you?

The clock sync tool ntpdate works fine for me to solve this problem. You can Google it for more details.

[sanha9999]

hi. @nate-wandb @vanpelt , I synchronized the linux server time using rdate, but the error is still the same. Is there any other solution?

[vanpelt]

Can you share the exact error you're seeing? Does it say "certificate is not yet valid"?

[sanha9999]

at the linux terminal

wandb login

wandb: Network error (SSLError), entering retry loop.
wandb: W&B API key is configured. Use `wandb login --relogin` to force relogin

[jmjeon2]

@jjahanip It worked for me. Thanks a lot!

[zhimin-z]

The error looks like clock skew. Can you be sure the time is correct on whatever server you're connecting from?

Hi @vanpelt Do you know how to deal with it? I have a similar ssl connection issue and tried pip install pip-system-certs according to @jjahanip but the error still persists.

[heyuepeng]

I have the same problem

[amazing-cc]

It looks like a network problem. My solution to this problem is to use a proxy. You need to configure your proxy into the runtime environment variables. Assuming you are using pycharm to execute your code, you can find Edit configurations in the top right corner and fill in the proxy configuration into the environment variable. Like this: HTTP_PROXY=http://127.0.0.1:7890;HTTPS_PROXY=http://127.0.0.1:7890 Then you can click run to run the training script. If you are using another IDE, I believe there are similar configurations. Before doing this, make sure you enable your proxy server, such as Clash

I have the same problem