Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)

[Meorge]

I'm using the following script to try to train the 124M model on a dataset.

from aitextgen import aitextgen
from os import environ

dataset = "dataset.csv"

def train():
    ai = aitextgen(tf_gpt2="124M", to_gpu=False)
    ai.train(dataset,
        line_by_line=True,
        from_cache=False,
        batch_size=1,
        num_steps=50,
        generate_every=5,
        save_every=10,
        progress_bar_refresh_rate=1)


if __name__ == "__main__":
    environ["TOKENIZERS_PARALLELISM"] = "false"
    train()

When I run the script, I get the following output:

> python3.7 generate.py
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2996/2996 [00:00<00:00, 16792.42it/s]
GPU available: False, used: False
TPU available: None, using: 0 TPU cores
  0%|                                                                                                                                                             | 0/50 [00:00<?, ?it/s]
[W ParallelNative.cpp:206] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
[W ParallelNative.cpp:206] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
[W ParallelNative.cpp:206] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
[W ParallelNative.cpp:206] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
[W ParallelNative.cpp:206] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
[W ParallelNative.cpp:206] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
[W ParallelNative.cpp:206] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
[W ParallelNative.cpp:206] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
Loss: 3.930 — Avg: 2.798:   6%|███████▍                                                                                                                   | 3/50 [00:39<10:24, 13.29s/it]

It doesn't seem like it's causing the program to actually behave incorrectly, but the warning still makes me worry something may be going wrong. I found online that including environ["OMP_NUM_THREADS"] = "1" suppresses this error, but according to them, it doesn't fix the underlying issue.

[minimaxir]

Warnings are fine, albeit not ideal. Training tends to generate a lot of them, although this particular warning I have not seen before (it's likely pytorch-lightning's fault and nothing that can be controlled with aitextgen).

What OS/CPU are you using?

[Meorge]

Alright, that's good to hear, thanks! I'm running macOS Big Sur 11.2.1 on a 3.8 GHz 8-Core Intel Core i7 processor

[minimaxir]

Weird, I have a similar setup and have not hit such warnings, although I have not set environ["TOKENIZERS_PARALLELISM"] = "false"