program frozen

[dlnewbie-h]

Dear authors

Your work is very exciting and I want to try out your code. I followed the instructions on readme, and is trying to run this example

CUDA_VISIBLE_DEVICES=0 python fixmatch.py --filters=32 --dataset=cifar10.3@40-1 --train_dir ./experiments/fixmatch

however my program get stucked at self.train_step forever...

I did installed the required environments as you pointed out in the readme.

Do you have any idea what's going on?

I

[dlnewbie-h]

after i keyboard interrupt it, it gives following message:

KeyboardInterrupt Traceback (most recent call last): File "xxxxxx/miniconda2/envs/fixmatch_env/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "xxxxxx/miniconda2/envs/fixmatch_env/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "xxxxxx/miniconda2/envs/fixmatch_env/lib/python3.7/multiprocessing/pool.py", line 110, in worker task = get() File "xxxxxx/miniconda2/envs/fixmatch_env/lib/python3.7/multiprocessing/queues.py", line 351, in get with self._rlock: File "xxxxxx/miniconda2/envs/fixmatch_env/lib/python3.7/multiprocessing/synchronize.py", line 95, in enter return self._semlock.enter() KeyboardInterrupt ^CProcess ForkPoolWorker-868: Traceback (most recent call last): File "xxxxxx/miniconda2/envs/fixmatch_env/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "xxxxxx/miniconda2/envs/fixmatch_env/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "xxxxxx/miniconda2/envs/fixmatch_env/lib/python3.7/multiprocessing/pool.py", line 110, in worker task = get() File "xxxxxx/miniconda2/envs/fixmatch_env/lib/python3.7/multiprocessing/queues.py", line 351, in get with self._rlock: File "xxxxxx/miniconda2/envs/fixmatch_env/lib/python3.7/multiprocessing/synchronize.py", line 95, in enter return self._semlock.enter() KeyboardInterrupt

[carlini]

Is the GPU active when you're running the training? Or is that stalling too?

[dlnewbie-h]

Is the GPU active when you're running the training? Or is that stalling too?

The GPU is stalling too... It's a Tesla V100, the CUDA version is 11.0 and Driver version 450.80.02 (btw I installed the specified requirements using conda, is this causing the issue?) Thank you

[carlini]

I just wanted to follow up to see if you worked anything out. I don't know if I have any ideas for what could be causing this with our code, but maybe you found a problem?

[dlnewbie-h]

Hi Carlini

Really thankful that you followed up on the issue. I really have trouble figuring out the issue which prevents me from using the code.

I first suspect it is some environment issue, here is the environment i'm using: (I think i matched your specified requirements? I tried the code on both tesla a100 and p100, and still don't work). I would really be helpful if you are able to help me with it.

This file may be used to create an environment using:

$ conda create --name --file

platform: linux-64

_libgcc_mutex=0.1=main _tflow_select=2.1.0=gpu absl-py=0.11.0=pyhd3eb1b0_1 astor=0.8.1=py37h06a4308_0 blas=1.0=mkl c-ares=1.17.1=h27cfd23_0 ca-certificates=2021.1.19=h06a4308_0 certifi=2020.12.5=py37h06a4308_0 cudatoolkit=10.1.243=h6bb024c_0 cudnn=7.6.5=cuda10.1_0 cupti=10.1.168=0 cycler=0.10.0=pypi_0 cython=0.29.21=py37h2531618_0 easydict=1.9=pypi_0 gast=0.4.0=py_0 google-pasta=0.2.0=py_0 grpcio=1.31.0=py37hf8bcb03_0 h5py=2.10.0=py37hd6299e0_1 hdf5=1.10.6=hb1b8bf9_0 importlib-metadata=2.0.0=py_1 intel-openmp=2020.2=254 keras-applications=1.0.8=py_1 keras-preprocessing=1.1.2=pyhd3eb1b0_0 kiwisolver=1.3.1=pypi_0 ld_impl_linux-64=2.33.1=h53a641e_7 libedit=3.1.20191231=h14c3975_1 libffi=3.3=he6710b0_2 libgcc-ng=9.1.0=hdf63c60_0 libgfortran-ng=7.3.0=hdf63c60_0 libprotobuf=3.14.0=h8c45485_0 libstdcxx-ng=9.1.0=hdf63c60_0 markdown=3.3.3=py37h06a4308_0 matplotlib=3.3.4=pypi_0 mkl=2020.2=256 mkl-service=2.3.0=py37he8ac12f_0 mkl_fft=1.2.0=py37h23d657b_0 mkl_random=1.1.1=py37h0573a6f_0 ncurses=6.2=he6710b0_1 numpy=1.19.2=py37h54aff64_0 numpy-base=1.19.2=py37hfa32c7d_0 opencv-python=4.5.1.48=pypi_0 openssl=1.1.1i=h27cfd23_0 pandas=1.2.1=py37ha9443f7_0 pillow=8.1.0=pypi_0 pip=20.3.3=py37h06a4308_0 protobuf=3.14.0=py37h2531618_1 pyparsing=2.4.7=pypi_0 python=3.7.9=h7579374_0 python-dateutil=2.8.1=pyhd3eb1b0_0 pytz=2021.1=pyhd3eb1b0_0 readline=8.1=h27cfd23_0 scipy=1.6.0=py37h91f5cce_0 setuptools=52.0.0=py37h06a4308_0 six=1.15.0=py37h06a4308_0 sqlite=3.33.0=h62c20be_0 tensorboard=1.14.0=py37hf484d3e_0 tensorflow=1.14.0=gpu_py37h74c33d7_0 tensorflow-base=1.14.0=gpu_py37he45bfe2_0 tensorflow-estimator=1.14.0=py_0 tensorflow-gpu=1.14.0=h0d30ee6_0 termcolor=1.1.0=py37_1 tk=8.6.10=hbc83047_0 tqdm=4.56.2=pypi_0 werkzeug=1.0.1=pyhd3eb1b0_0 wheel=0.36.2=pyhd3eb1b0_0 wrapt=1.12.1=py37h7b6447c_1 xz=5.2.5=h7b6447c_0 zipp=3.4.0=pyhd3eb1b0_0 zlib=1.2.11=h7b6447c_3

[carlini]

Huh. Two ideas maybe:

1. Does CPU-only training work? It would be really slow, but it should at least not stall.

2. If you try to train a MixMatch model with the MixMatch codebase (https://github.com/google-research/mixmatch) does that work? It's very similar code, and this might help isolate if it's a problem with fixmatch or with the general environment.

[dlnewbie-h]

I followed your suggestion, which provide me with some useful insights. So

1. fixmatch code does work on cpu-only training (using another environment that has cpu version tensorflow).
2. I also tried the MixMatch model you pointed to. MixMatch work on both tesla p100, v100, but not a100. (using the gpu tensorflow environment i showed you above)

So I guess

1. there might be something unique to fixmatch that make fixmatch doesn't work on my p100 and v100 (while MixMatch works).
2. There must be something wrong with my a100s.

For my concern now, I would be very happy if i can get fixmatch to work on my p100 or v100. I really don't understand what could have happened that make mixmatch able to run but fixmatch not able to run? Do you have any thoughts?

[carlini]

That's very interesting. This fixmatch codebase has an implementation of mixmatch. Does that also run properly?

If that works, then maybe try running fixmatch with something like --uratio=1 and see if that helps. Maybe it's the batch size that's the problem?

[dlnewbie-h]

Hi Carlini

That's very weird, i cannot run the implementation of mixmatch from the fixmatch codebase (while i'm able to run the original mixmatch codebase). May i ask what GPU do you run the fixmatch codebase on? could it be the new code not compatible with some GPUs? (I have very limited knowledge about this, hope that it is not a ridiculous guess...)

[carlini]

That is very strange. We've never seen any issues with different GPUs in the past, and the two codebases are very similar.

Maybe @david-berthelot has some insight that I'm missing.

[dlnewbie-h]

In your requirment.txt, there's no specific requirement for python version or cudatoolkit, cudnn, cuda version. Do I need to install any specific version of cudatoolkit, cudnn or cuda?