cdvae
cdvae copied to clipboard
Published
20 hours ago •
txie-93
txie-93
Code Hanging At Start of Training
Hi, I am running the CDVAE carbon experiment and I have been seeing a weird error. It appears that my code will just hang after completely three iterations of the first epoch.
I run **python cdvae/run.py data=carbon expname=carbon model.predict_property=True**
The output I see is this:
`[2023-07-13 16:57:36,190][hydra.utils][INFO] - Instantiating <cdvae.pl_data.datamodule.CrystDataModule>
[2023-07-13 16:57:37,161][numexpr.utils][INFO] - Note: detected 128 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
[2023-07-13 16:57:37,161][numexpr.utils][INFO] - Note: NumExpr detected 128 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
25%|█████████████████████▍ | 1521/6091 [00:25<01:29, 50.81it/s]/home/.conda/envs/cdvae/lib/python3.8/site-packages/pymatgen/io/cif.py:1120: UserWarning: Issues encountered while parsing CIF: Some fractional coordinates rounded to ideal values to avoid issues with finite precision.
warnings.warn("Issues encountered while parsing CIF: " + "\n".join(self.warnings))
34%|█████████████████████████████▎ | 2080/6091 [00:34<01:05, 61.51it/s]/home/.conda/envs/cdvae/lib/python3.8/site-packages/pymatgen/io/cif.py:1120: UserWarning: Issues encountered while parsing CIF: Some fractional coordinates rounded to ideal values to avoid issues with finite precision.
warnings.warn("Issues encountered while parsing CIF: " + "\n".join(self.warnings))
46%|███████████████████████████████████████▊ | 2820/6091 [00:46<01:02, 52.70it/s]/home/.conda/envs/cdvae/lib/python3.8/site-packages/pymatgen/io/cif.py:1120: UserWarning: Issues encountered while parsing CIF: Some fractional coordinates rounded to ideal values to avoid issues with finite precision.
warnings.warn("Issues encountered while parsing CIF: " + "\n".join(self.warnings))
50%|██████████████████████████████████████████▋ | 3021/6091 [00:49<00:52, 58.26it/s]/home/.conda/envs/cdvae/lib/python3.8/site-packages/pymatgen/io/cif.py:1120: UserWarning: Issues encountered while parsing CIF: Some fractional coordinates rounded to ideal values to avoid issues with finite precision.
warnings.warn("Issues encountered while parsing CIF: " + "\n".join(self.warnings))
51%|███████████████████████████████████████████▍ | 3079/6091 [00:50<00:54, 55.41it/s]/home/.conda/envs/cdvae/lib/python3.8/site-packages/pymatgen/io/cif.py:1120: UserWarning: Issues encountered while parsing CIF: Some fractional coordinates rounded to ideal values to avoid issues with finite precision.
warnings.warn("Issues encountered while parsing CIF: " + "\n".join(self.warnings))
51%|████████████████████████████████████████████▏ | 3132/6091 [00:51<00:40, 72.78it/s]/home/.conda/envs/cdvae/lib/python3.8/site-packages/pymatgen/io/cif.py:1120: UserWarning: Issues encountered while parsing CIF: Some fractional coordinates rounded to ideal values to avoid issues with finite precision.
warnings.warn("Issues encountered while parsing CIF: " + "\n".join(self.warnings))
52%|████████████████████████████████████████████▎ | 3140/6091 [00:51<00:49, 59.77it/s]/home/.conda/envs/cdvae/lib/python3.8/site-packages/pymatgen/io/cif.py:1120: UserWarning: Issues encountered while parsing CIF: Some fractional coordinates rounded to ideal values to avoid issues with finite precision.
warnings.warn("Issues encountered while parsing CIF: " + "\n".join(self.warnings))
60%|███████████████████████████████████████████████████▊ | 3673/6091 [00:59<00:38, 63.39it/s]/home/.conda/envs/cdvae/lib/python3.8/site-packages/pymatgen/io/cif.py:1120: UserWarning: Issues encountered while parsing CIF: Some fractional coordinates rounded to ideal values to avoid issues with finite precision.
warnings.warn("Issues encountered while parsing CIF: " + "\n".join(self.warnings))
66%|████████████████████████████████████████████████████████▋ | 4018/6091 [01:05<00:32, 63.75it/s]/home/.conda/envs/cdvae/lib/python3.8/site-packages/pymatgen/io/cif.py:1120: UserWarning: Issues encountered while parsing CIF: Some fractional coordinates rounded to ideal values to avoid issues with finite precision.
warnings.warn("Issues encountered while parsing CIF: " + "\n".join(self.warnings))
67%|█████████████████████████████████████████████████████████▌ | 4077/6091 [01:06<00:33, 60.74it/s]/home/.conda/envs/cdvae/lib/python3.8/site-packages/pymatgen/io/cif.py:1120: UserWarning: Issues encountered while parsing CIF: Some fractional coordinates rounded to ideal values to avoid issues with finite precision.
warnings.warn("Issues encountered while parsing CIF: " + "\n".join(self.warnings))
67%|█████████████████████████████████████████████████████████▊ | 4098/6091 [01:06<00:29, 67.92it/s]/home/.conda/envs/cdvae/lib/python3.8/site-packages/pymatgen/io/cif.py:1120: UserWarning: Issues encountered while parsing CIF: Some fractional coordinates rounded to ideal values to avoid issues with finite precision.
warnings.warn("Issues encountered while parsing CIF: " + "\n".join(self.warnings))
69%|███████████████████████████████████████████████████████████▊ | 4233/6091 [01:08<00:29, 63.53it/s]/home/.conda/envs/cdvae/lib/python3.8/site-packages/pymatgen/io/cif.py:1120: UserWarning: Issues encountered while parsing CIF: Some fractional coordinates rounded to ideal values to avoid issues with finite precision.
warnings.warn("Issues encountered while parsing CIF: " + "\n".join(self.warnings))
74%|████████████████████████████████████████████████████████████████ | 4536/6091 [01:13<00:23, 67.50it/s]/home/.conda/envs/cdvae/lib/python3.8/site-packages/pymatgen/io/cif.py:1120: UserWarning: Issues encountered while parsing CIF: Some fractional coordinates rounded to ideal values to avoid issues with finite precision.
warnings.warn("Issues encountered while parsing CIF: " + "\n".join(self.warnings))
80%|████████████████████████████████████████████████████████████████████▋ | 4869/6091 [01:18<00:16, 72.17it/s]/home/.conda/envs/cdvae/lib/python3.8/site-packages/pymatgen/io/cif.py:1120: UserWarning: Issues encountered while parsing CIF: Some fractional coordinates rounded to ideal values to avoid issues with finite precision.
warnings.warn("Issues encountered while parsing CIF: " + "\n".join(self.warnings))
84%|████████████████████████████████████████████████████████████████████████ | 5106/6091 [01:22<00:18, 53.65it/s]/home/.conda/envs/cdvae/lib/python3.8/site-packages/pymatgen/io/cif.py:1120: UserWarning: Issues encountered while parsing CIF: Some fractional coordinates rounded to ideal values to avoid issues with finite precision.
warnings.warn("Issues encountered while parsing CIF: " + "\n".join(self.warnings))
91%|██████████████████████████████████████████████████████████████████████████████▌ | 5566/6091 [01:29<00:08, 63.96it/s]/home/.conda/envs/cdvae/lib/python3.8/site-packages/pymatgen/io/cif.py:1120: UserWarning: Issues encountered while parsing CIF: Some fractional coordinates rounded to ideal values to avoid issues with finite precision.
warnings.warn("Issues encountered while parsing CIF: " + "\n".join(self.warnings))
95%|█████████████████████████████████████████████████████████████████████████████████▋ | 5786/6091 [01:33<00:05, 59.95it/s]/home/.conda/envs/cdvae/lib/python3.8/site-packages/pymatgen/io/cif.py:1120: UserWarning: Issues encountered while parsing CIF: Some fractional coordinates rounded to ideal values to avoid issues with finite precision.
warnings.warn("Issues encountered while parsing CIF: " + "\n".join(self.warnings))
96%|██████████████████████████████████████████████████████████████████████████████████▏ | 5822/6091 [01:33<00:04, 64.72it/s]/home/.conda/envs/cdvae/lib/python3.8/site-packages/pymatgen/io/cif.py:1120: UserWarning: Issues encountered while parsing CIF: Some fractional coordinates rounded to ideal values to avoid issues with finite precision.
warnings.warn("Issues encountered while parsing CIF: " + "\n".join(self.warnings))
98%|████████████████████████████████████████████████████████████████████████████████████▎ | 5974/6091 [01:36<00:01, 66.91it/s]/home/.conda/envs/cdvae/lib/python3.8/site-packages/pymatgen/io/cif.py:1120: UserWarning: Issues encountered while parsing CIF: Some fractional coordinates rounded to ideal values to avoid issues with finite precision.
warnings.warn("Issues encountered while parsing CIF: " + "\n".join(self.warnings))
100%|██████████████████████████████████████████████████████████████████████████████████████| 6091/6091 [01:39<00:00, 61.48it/s]
/gpfs/fs1/home/cdvae-old/cdvae/cdvae/common/data_utils.py:644: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /data/miniconda3/envs/opence-1.7/conda-bld/pytorch-base_1663986328871/work/torch/csrc/utils/tensor_new.cpp:201.)
targets = torch.tensor([d[key] for d in data_list])
/gpfs/fs1/home/cdvae-old/cdvae/cdvae/common/data_utils.py:612: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
X = torch.tensor(X, dtype=torch.float)
[2023-07-13 16:59:20,540][hydra.utils][INFO] - Instantiating <cdvae.pl_modules.model.CDVAE>
[2023-07-13 16:59:20,615][torch.distributed.nn.jit.instantiator][INFO] - Created a temporary directory at /tmp/tmpwv1glt9u
[2023-07-13 16:59:20,615][torch.distributed.nn.jit.instantiator][INFO] - Writing /tmp/tmpwv1glt9u/_remote_module_non_scriptable.py
[2023-07-13 16:59:53,346][hydra.utils][INFO] - Passing scaler from datamodule to model <StandardScalerTorch(means: -154.2510223388672, stds: 0.13738815486431122)>
[2023-07-13 16:59:53,348][hydra.utils][INFO] - Adding callback <LearningRateMonitor>
[2023-07-13 16:59:53,349][hydra.utils][INFO] - Adding callback <EarlyStopping>
[2023-07-13 16:59:53,350][hydra.utils][INFO] - Adding callback <ModelCheckpoint>
[2023-07-13 16:59:53,354][hydra.utils][INFO] - Instantiating <WandbLogger>
wandb: Currently logged in as: _. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.15.5
wandb: Run data is saved locally in /home/cdvae-old/cdvae/wabdb/wandb/run-20230713_165954-u04zv43g
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run carbon
wandb: ⭐️ View project at https://wandb.ai/_/crystal_generation_mit
wandb: 🚀 View run at https://wandb.ai/_/crystal_generation_mit/runs/u04zv43g
[2023-07-13 17:00:07,550][hydra.utils][INFO] - W&B is now watching <{cfg.logging.wandb_watch.log}>!
wandb: logging graph, to disable use `wandb.watch(log_graph=False)`
[2023-07-13 17:00:07,588][hydra.utils][INFO] - Instantiating the Trainer
/home/.conda/envs/cdvae/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/callback_connector.py:96: LightningDeprecationWarning: Setting `Trainer(progress_bar_refresh_rate=20)` is deprecated in v1.5 and will be removedin v1.7. Please pass `pytorch_lightning.callbacks.progress.TQDMProgressBar` with `refresh_rate` directly to the Trainer's `callbacks` argument instead. Or, to disable the progress bar pass `enable_progress_bar = False` to the Trainer.
rank_zero_deprecation(
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
[2023-07-13 17:00:07,650][hydra.utils][INFO] - Starting training!
0%| | 2/6091 [00:00<32:19, 3.14it/s
I am running on MIST HPC, so I have turned off WandB logging.
Environment
Package Version Editable project location
------------------------ ----------------- --------------------------------------------------
absl-py 1.4.0
aiofiles 22.1.0
aiohttp 3.8.4
aiosignal 1.3.1
aiosqlite 0.18.0
altair 5.0.1
antlr4-python3-runtime 4.8
anyio 3.5.0
appdirs 1.4.4
argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0
ase 3.22.0
astor 0.8.1
astroid 2.14.2
asttokens 2.0.5
async-timeout 4.0.2
attrs 22.1.0
autopep8 2.0.2
av 9.2.0
Babel 2.11.0
backcall 0.2.0
backports.zoneinfo 0.2.1
base58 2.1.1
beautifulsoup4 4.12.2
bleach 4.1.0
blinker 1.6.2
Bottleneck 1.3.5
brotlipy 0.7.0
cachetools 5.3.1
cdvae 0.0.1
certifi 2023.5.7
cffi 1.15.1
charset-normalizer 2.0.4
click 8.0.4
colorama 0.4.6
comm 0.1.2
configparser 6.0.0
contourpy 1.0.5
coverage 7.2.2
cryptography 39.0.1
cycler 0.11.0
debugpy 1.5.1
decorator 5.1.1
defusedxml 0.7.1
dill 0.3.6
distlib 0.3.6
dnspython 2.3.0
docker-pycreds 0.4.0
emmet-core 0.60.1
entrypoints 0.4
exceptiongroup 1.0.4
executing 0.8.3
fastjsonschema 2.16.2
filelock 3.12.0
fonttools 4.25.0
frozenlist 1.3.3
fsspec 2023.4.0
future 0.18.3
gitdb 4.0.10
GitPython 3.1.32
google-auth 2.22.0
google-auth-oauthlib 1.0.0
googledrivedownloader 0.4
grpcio 1.48.2
higher 0.2.1
html5lib 1.1
hydra-core 1.1.0
hydra-joblib-launcher 1.1.5
idna 3.4
importlib-metadata 6.0.0
importlib-resources 5.12.0
iniconfig 1.1.1
ipykernel 6.19.2
ipython 8.12.0
ipython-genutils 0.2.0
ipywidgets 8.0.4
isodate 0.6.1
isort 5.9.3
jedi 0.18.1
Jinja2 3.1.2
joblib 1.2.0
json5 0.9.6
jsonschema 4.17.3
jupyter_client 8.1.0
jupyter_core 5.3.0
jupyter-events 0.6.3
jupyter_server 2.5.0
jupyter_server_fileid 0.9.0
jupyter_server_terminals 0.4.4
jupyter_server_ydoc 0.8.0
jupyter-ydoc 0.2.4
jupyterlab 3.6.3
jupyterlab-pygments 0.1.2
jupyterlab_server 2.22.0
jupyterlab-widgets 3.0.5
kiwisolver 1.4.4
latexcodec 2.0.1
lazy-object-proxy 1.6.0
lightning-utilities 0.7.1
lxml 4.9.2
Markdown 3.4.3
MarkupSafe 2.1.1
matminer 0.7.3
matplotlib 3.7.1
matplotlib-inline 0.1.6
mccabe 0.7.0
mistune 0.8.4
monty 2023.5.8
mp-api 0.33.3
mpmath 1.3.0
msgpack 1.0.5
multidict 6.0.4
multiprocess 0.70.14
munkres 1.1.4
nbclassic 0.5.5
nbclient 0.5.13
nbconvert 6.5.4
nbformat 5.7.0
nest-asyncio 1.5.6
networkx 2.8.4
nglview 3.0.6
notebook 6.5.4
notebook_shim 0.2.2
numexpr 2.8.4
numpy 1.23.5
oauthlib 3.2.2
omegaconf 2.1.2
p-tqdm 1.3.3
packaging 23.0
palettable 3.3.3
pandas 1.5.3
pandocfilters 1.5.0
parso 0.8.3
pathos 0.3.0
pathtools 0.1.2
pexpect 4.8.0
pickleshare 0.7.5
Pillow 9.4.0
Pint 0.21.1
pip 23.1.2
pkgutil_resolve_name 1.3.10
platformdirs 3.2.0
plotly 5.15.0
pluggy 1.0.0
pox 0.3.2
ppft 1.7.6.6
prometheus-client 0.14.1
promise 2.3
prompt-toolkit 3.0.36
protobuf 3.19.6
psutil 5.9.0
ptyprocess 0.7.0
pure-eval 0.2.2
py 1.11.0
pyarrow 8.0.0
pyasn1 0.5.0
pyasn1-modules 0.3.0
pybtex 0.24.0
pycodestyle 2.10.0
pycparser 2.21
pydantic 1.10.11
pydeck 0.8.1b0
pyDeprecate 0.3.1
pyg-nightly 2.4.0.dev20230711
Pygments 2.15.1
pylint 2.16.2
pymatgen 2023.7.11
pymongo 4.4.0
pyOpenSSL 23.0.0
pyparsing 3.0.9
pyrsistent 0.18.0
PySocks 1.7.1
pytest 7.3.1
pytest-cov 4.0.0
python-dateutil 2.8.2
python-dotenv 1.0.0
python-json-logger 2.0.7
python-louvain 0.15
pytorch-lightning 1.6.5
pytz 2022.7
PyYAML 5.4.1
pyzmq 25.1.0
rdflib 6.1.1
requests 2.29.0
requests-oauthlib 1.3.1
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rsa 4.9
ruamel.yaml 0.17.32
ruamel.yaml.clib 0.2.7
scikit-learn 1.2.2
scipy 1.8.1
Send2Trash 1.8.0
sentencepiece 0.1.96
sentry-sdk 1.28.0
setproctitle 1.3.2
setuptools 67.8.0
shortuuid 1.0.11
six 1.16.0
SMACT 2.2.1
smmap 5.0.0
sniffio 1.2.0
soupsieve 2.4
spglib 2.0.2
stack-data 0.2.0
streamlit 0.79.0
subprocess32 3.5.4
sympy 1.12
tabulate 0.8.10
tenacity 8.2.2
tensorboard 2.13.0
tensorboard-data-server 0.7.1
terminado 0.17.1
threadpoolctl 2.2.0
tinycss2 1.2.1
toml 0.10.2
tomli 2.0.1
tomlkit 0.11.1
toolz 0.12.0
torch 1.12.1
torch-cluster 1.6.1
torch-geometric 1.7.2
torch-scatter 2.0.8
torch-sparse 0.6.10
torch-spline-conv 1.2.2
torchdiffeq 0.0.1
torchmetrics 1.0.0
torchtext 0.13.1a0+35066f2
torchvision 0.13.1
tornado 6.2
tqdm 4.65.0
traitlets 5.7.1
typing_extensions 4.6.3
tzlocal 5.0.1
uncertainties 3.1.7
urllib3 1.26.16
validators 0.20.0
virtualenv 20.22.0
wandb 0.15.5
watchdog 3.0.0
wcwidth 0.2.5
webencodings 0.5.1
websocket-client 0.58.0
Werkzeug 2.3.6
wheel 0.38.4
widgetsnbextension 4.0.5
wrapt 1.14.1
y-py 0.5.9
yacs 0.1.6
yarl 1.9.2
ypy-websocket 0.8.2
zipp 3.11.0
Any suggestions on how to resolve this? I am not very familiar with Hydra and Pytorch Lightning.
