Hi, I’m using comel-ml in docker container. The container is connected to the Internet, but it can’t upload and receive the response.

[G-B-Y]

Hi, I’m using comel-ml in docker container. The container is connected to the Internet, but it can’t upload and receive the response. The container has connected to the internet, but it can't upload the data. I can ping www.google.com. And in windows, the data can be uploaded, but in container can't. Ubuntu16.04, Python3.7, PyTorch1.6, cuda10.1, cudnn7, comet-ml3.21.0.

[DN6]

Hello @G-B-Y. Could you ssh into your container, run the following command inside the container, and share the output with me here?

comet check --debug

[G-B-Y]

Hi @DN6, thank you for answer. Here is the result.

[G-B-Y]

In container it still stucks, and can't be uploaded.

[G-B-Y]

COMET INFO: Uploads: COMET INFO: environment details : 1 COMET INFO: filename : 1 COMET INFO: installed packages : 1 COMET INFO: model graph : 1 COMET INFO: os packages : 1 COMET INFO: source_code : 2 (13.91 KB) COMET INFO: --------------------------- COMET ERROR: Error sending a notification, make sure you have opted-in for notifications COMET INFO: Uploading metrics, params, and assets to Comet before program termination (may take several seconds) COMET INFO: The Python SDK has 3600 seconds to finish before aborting... COMET INFO: Uploading 3176 metrics, params and output messages COMET INFO: Uploading 3176 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET ERROR: File could not be uploaded COMET ERROR: File could not be uploaded COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET WARNING: Failed to send all messages, metrics and output will likely be incomplete. COMET ERROR: Comet failed to send all the data back (1851 messages and 0 uploads) COMET WARNING: Failed to log run in comet.ml

[DN6]

Hi @G-B-Y. It might be that the internet connection on the machine is not stable throughout the duration of the experiment. Could you try using the OfflineExperiment?

This means that you would change your experiment creation step from

experiment = comet_ml.Experiment()

to

experiment = comet_ml.OfflineExperiment(offline_directory='./')

This will save the experiment as a zip file to the current working directory in your container. You can then upload this experiment using

comet upload ./<Your Experiment Key>.zip

Could you test to see if this is working for you?

[G-B-Y]

Hi, thank you for answer. I try the way you suggest, but it still can’t work. The result is as followed.

@.***:/workspace# comet upload ./comet_data/a119e1686f004de688873b4b0268e74d.zip COMET INFO: Attempting to upload './comet_data/a119e1686f004de688873b4b0268e74d.zip'... COMET INFO: Starting the upload of the experiment COMET ERROR: File could not be uploaded COMET ERROR: File could not be uploaded COMET WARNING: Error processing line 781 COMET WARNING: Error processing line 782 COMET WARNING: Error processing line 783

从 Windows 版邮件发送

发件人: Dhruv Nair 发送时间: 2021年11月13日 1:51 收件人: comet-ml/issue-tracking 抄送: G-B-Y; Mention 主题: Re: [comet-ml/issue-tracking] Hi, I’m using comel-ml in docker container. The container is connected to the Internet, but it can’t upload and receive the response. (Issue #444)

Hi @G-B-Y. It might be that the internet connection on the machine is not stable throughout the duration of the experiment. Could you try using the OfflineExperiment? This means that you would change your experiment creation step from experiment = comet_ml.Experiment() to experiment = comet_ml.OfflineExperiment(offline_directory='./') This will save the experiment as a zip file to the current working directory in your container. You can then upload this experiment using comet upload ./<Your Experiment Key>.zip Could you test to see if this is working for you? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[DN6]

Hello @G-B-Y. Sorry that did not work for you. Could you try saving the resulting offline experiment zip file directly on your machine, instead of inside the container? And try uploading from your machine?

[G-B-Y]

Hello, thank you for answer. I will try the way that you suggest, but it isn’t flexible any more.😢

从 Windows 版邮件发送

发件人: Dhruv Nair 发送时间: 2021年11月18日 0:37 收件人: comet-ml/issue-tracking 抄送: G-B-Y; Mention 主题: Re: [comet-ml/issue-tracking] Hi, I’m using comel-ml in docker container. The container is connected to the Internet, but it can’t upload and receive the response. (Issue #444)

Hello @G-B-Y. Sorry that did not work for you. Could you try saving the resulting offline experiment zip file directly on your machine, instead of inside the container? And try uploading from your machine? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[G-B-Y]

Hi, there is a question that may be important: could IP-“172.17.0.3” case the upload error?

从 Windows 版邮件发送

发件人: Dhruv Nair 发送时间: 2021年11月18日 0:37 收件人: comet-ml/issue-tracking 抄送: G-B-Y; Mention 主题: Re: [comet-ml/issue-tracking] Hi, I’m using comel-ml in docker container. The container is connected to the Internet, but it can’t upload and receive the response. (Issue #444)

Hello @G-B-Y. Sorry that did not work for you. Could you try saving the resulting offline experiment zip file directly on your machine, instead of inside the container? And try uploading from your machine? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[DN6]

Hi @G-B-Y. Were you able to upload the zip file from outside the container? I believe the IP you are referring to is one that is assigned to the containers on your machine's subnetwork.

[G-B-Y]

Hi, thank you for answer. I’m sorry that the docker outside the container still can’t upload. The container’s image is “1.9.0-cuda11.1-cudnn8-devel”.

从 Windows 版邮件发送

发件人: Dhruv Nair 发送时间: 2021年11月22日 23:34 收件人: comet-ml/issue-tracking 抄送: G-B-Y; Mention 主题: Re: [comet-ml/issue-tracking] Hi, I’m using comel-ml in docker container. The container is connected to the Internet, but it can’t upload and receive the response. (Issue #444)

Hi @G-B-Y. Were you able to upload the zip file from outside the container? I believe the IP you are referring to is one that is assigned to the containers on your machine's subnetwork. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[DN6]

Hi @G-B-Y. To clarify, you saved the zip file from the Offline Experiment outside the container on to your windows machine and then tried uploading it to Comet by running the following command outside the container on your windows machine?

comet upload <Path to your Experiment Key.zip file>

[G-B-Y]

Sorry for ambiguous expression. The container is the sub environment of the host. The host system is Ubuntu LTX 16.04.7, the container is image of "pytorch/pytorch:1.9.0-cuda11.1-cudnn8-devel", and there is no windows machine.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.