openvino_notebooks icon indicating copy to clipboard operation
openvino_notebooks copied to clipboard

Add NNCF PyTorch INT8 quantization notebook

Open vshampor opened this issue 3 years ago • 3 comments

vshampor avatar Aug 16 '21 13:08 vshampor

Cell 12: Timeout of 2000 seconds exceeded while executing cell. Failed to interrupt kernel in 5 seconds, so failing without traceback.

Looks like it got timeout... Would you like first cross-check that?

raymondlo84 avatar Aug 19 '21 15:08 raymondlo84

Cell 12: Timeout of 2000 seconds exceeded while executing cell. Failed to interrupt kernel in 5 seconds, so failing without traceback.

Looks like it got timeout... Would you like first cross-check that?

Can't be related to any errors on the part of the notebook, since no Python-specific code is present and builds for all the other python versions are able to complete within reasonable time.

Cell 12 (if I'm counting correctly) is the most time-intensive cell which actually launches PyTorch-powered model fine-tuning, so it is expected to take quite some time. I wonder if the executor node for the build even has GPUs, and if not (in which case the CPU powered training is used), whether the executor had not been starved for CPU resources by other jobs on the same machine.

Either way we won't get a direct answer since the hosts are Github's (correct me if I'm wrong), so is there a way to simply restart the builds and see if this is indeed a stable issue?

vshampor avatar Aug 19 '21 16:08 vshampor

@helena-intel could you please assist with interpreting the CI failures?

vshampor avatar Sep 15 '21 09:09 vshampor

After our internal conversation, I will close this PR. Again a thousand apologies to the contributor. Next time, we will have more assertive communication. We are working to reduce the processing time with each PR. Thanks and kind regards.

paularamo avatar Dec 15 '22 13:12 paularamo