RuntimeError: Failed to import transformers.optimization

[riatzukiza]

Describe the bug I've downloaded the corpus and the model weights, I ran the command bash training/finetune_GPT-NeoXT-Chat-Base-20B.sh and I got the following: https://gist.github.com/riatzukiza/0930307fc90bf940103364be2d3db5c1

To Reproduce Steps to reproduce the behavior:

1. Download weights
2. download corpus
3. run bash training/finetune_GPT-NeoXT-Chat-Base-20B.sh
4. Bam error

Expected behavior

To fine tune the model, or get an out of memory error

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Pop os

Additional context Add any other context about the problem here.

[orangetin]

Looking at your error log, this seems to be important:

RuntimeError: Failed to import transformers.optimization because of the following error (look up to see its traceback):
Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

• What version of protoc do you have?
• Can you just try importing transformers.optimization in a Python script and see if it gives you any errors?
• Can you try running this line in a Python script and see if you get any errors: from transformers import get_linear_schedule_with_warmup

Honestly, it looks like it may be a package or env thing. Check to make sure your conda env is set up properly.

The error comes from line 12 of training/pipeline_parallel/dist_gpipe_pipeline_async.py. Can't proceed without more info.

[csris]

I'm not sure, but I agree with @orangetin that the environment might not be set up properly. Did you install packages using environment.yml at the root of the repo?

[riatzukiza]

I used the environment.yml like the readme said, it did take a while. I'm taking a look at it again today/tomarrow

[orangetin]

@riatzukiza Try the method from this comment if the installation takes too long. Mamba can be a lot faster.

[joecodecreations]

had the same issue. Try running this pip3 install --upgrade protobuf==3.20.0 to solve it