GraphQL 500 error with W&B behind Traefik

[Telcrome]

I am facing POST https://tls.reverseproxy.org/graphql2 500 (Internal Server Error) in the browser console when trying to access loggings such as confusion matrices with my W&B instance which is served behind a reverse proxy. Is there any way to view the server logs or let W&B correctly know it is accessed via another name?

Currently, I am trying to using the environment variable HOST=https://tls.reverseproxy.org/ on server side for that, which seems to not work.

To reproduce, log a confusion matrix:

import wandb
wandb.init() # I set correct host and token as environment variable

probs = [[0.1, 0.9], [0.8, 0.2]]
y_true = [0, 1]
wandb.log({"confmat": wandb.plot.confusion_matrix(probs, y_true)})

Observed in version: 0.66.0 & 0.66.3

[ArtsiomWB]

Hey @Telcrome! Thank you for writing in!

Could you please send us the Debug Bundle? The bundle is the server-side logs the user can access on their side. An admin of the instance can get it from the /system-admin page -> top right corner W&B icon -> Debug Bundle.

Per our docs, you could also try setting:

WANDB_BASE_URL	If you’re using wandb/local you should set this environment variable to http://YOUR_IP:YOUR_PORT

[Telcrome]

Thanks @ArtsiomWB for the quick reply! I have sent you an invitation to a repo where I have uploaded the debug log.

Is WANDB_BASE_URL an environment variable that is only used on the client side? It had no effect when set on the wandb/local side. Also, all wandb.logs are already successfully stored on our wandb/local, but things like confusion matrices only show up in the browser when accessing the instance without reverse proxy.

Whenever a wandb.log does not show up in the browser the internal server error is displayed in the browser console.

[ArtsiomWB]

Thank you for the follow up as well.

Were you able to collect the debug bundle? It might have some more information on what is causing the error. If you have, could you please send it our way?

[Telcrome]

It helped finding the error. The SSL certificate was missing. After setting it via the environment variable SSL_CERT_FILE it works. Thanks!