OMP: Error #100: Fatal system error detected.

[sfxworks]

(base) root@anaconda-statefulset-0:~/flan-alpaca# python data_loading.py preprocess_alpaca \
--path_in data/alpaca_clean.json \
--path_out data/train.json
OMP: Error #100: Fatal system error detected.
OMP: System error #22: Invalid argument
Aborted (core dumped)

I'm inside a k8s pod when doing this, but doing export KMP_AFFINITY=disabled resolved this for me. Not sure if it's being in a container or more. Using the anacodna3 container

[chiayewken]

Sorry, I'm not familiar with k8s pods, it could be an environment issue. It should not be a memory issue as the dataset is relatively small (less than 50MB)