Pytorch/XLA nightly version do not work on google colab: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found

[SankhaJay]

🐛 Bug

I want to use PyTorch/XLA nightly version on my project. But it does not work on google colab. Getting the error when importing torch

To Reproduce

VERSION = "nightly"  #@param ["1.5" , "20200325", "nightly"]
!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py
!python pytorch-xla-env-setup.py --version $VERSION

!git clone https://github.com/huggingface/transformers.git
!pip install ./transformers
!pip install -U nlp

when I simply write import torch it gives the following error:

ImportError                               Traceback (most recent call last)

<ipython-input-9-caf30418ab08> in <module>()
----> 1 import torch
      2 import nlp
      3 from transformers import T5Tokenizer

/usr/local/lib/python3.7/dist-packages/torch/__init__.py in <module>()
    195     if USE_GLOBAL_DEPS:
    196         _load_global_deps()
--> 197     from torch._C import *  # noqa: F403
    198 
    199 # Appease the type checker; ordinarily this binding is inserted by the

ImportError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch_cpu.so)

Additional context

I am using the same code mentioned in the following notebook https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb

[JackCaoG]

@SankhaJay Thanks for reporting! I think this has to do we recently switch to our wheel build from debian9 to debian10. We will look into how to unblock you. In the mean time, can you manually choose a slight older nightly. For example, use 20211010 as Version

[SankhaJay]

@JackCaoG thanks for the reply and suggestion. And it works for the import error.

[lihan]

Same problem today.

[JackCaoG]

We have some trouble building the colab compatible wheel due to recent build image update. This affect 1.10 and nightly. We are still working on it.

[Arka161]

+1, getting the same issue if I run the following command for installing Torch XLA on Colab:

!pip install cloud-tpu-client==0.10 torch==1.9.0 https://storage.googleapis.com/tpu-pytorch/wheels/torch_xla-1.10-cp37-cp37m-linux_x86_64.whl

[JackCaoG]

Hi @Arka161 , 1.10 is not supported right now. You are installing pytroch 1.9 and torch_xla 1.10 which will not work. Can you use torch_xla 1.9 in the mean time? Thanks.

[cajoek]

Hi @JackCaoG is pytorch 1.10 supported yet? Latest version says it does support it but I can't get it to work on Colab. What command should I use to install?

[JackCaoG]

@cajoek We didn't make much progress on 1.10(and nightly) on colab yet. It is moving pretty slowly.

[cajoek]

I see, are the problems specifically related to Colab?

[JackCaoG]

@cajoek It is related to colab's base os(I think it is ubuntu 18.02). PyTorch/Xla has moved its build to debian 10(to fix a different build issue) which has a newer version of GLIBC and incompatible with ubuntu. While colab team is working on upgrading their base os to a newer version and we are trying to build pt/xla on debian 9 too, both of these efforts goes pretty slowly.

[cajoek]

Hi @JackCaoG, has any progress been made on 1.10 for colab? Can I track the progress somewhere?

[Pasinduekanayake]

Hi @JackCaoG any updates on 1.10 for colab?

@cajoek It is related to colab's base os(I think it is ubuntu 18.02). PyTorch/Xla has moved its build to debian 10(to fix a different build issue) which has a newer version of GLIBC and incompatible with ubuntu. While colab team is working on upgrading their base os to a newer version and we are trying to build pt/xla on debian 9 too, both of these efforts goes pretty slowly.

[zirismetov]

+1 got same error on google COLAB . To fix i removed "import torch" ran the code(got the error that package is missing), and again imported it. Worked for me

[JackCaoG]

Hey sorry. We are almost there.. We fixed the build issue internally and will build a wheel for 1.10 and update the instruction hopefully next week. @yeounoh

[yeounoh]

Hi @SankhaJay , could you try re-installing torch torchvision torch-xla with the following commands on your colab notebook?

!pip install --no-cache cloud-tpu-client==0.10 torch==1.10 https://storage.googleapis.com/tpu-pytorch/wheels/colab/1.10/torch_xla-1.10-cp37-cp37m-linux_x86_64.whl

The torch_xla==1.10 wheel supports/requires torch==1.10 cc @Arka161 @Pasinduekanayake @cajoek

WARNING:root:Waiting for TPU to be start up with version pytorch-1.10...
WARNING:root:Waiting for TPU to be start up with version pytorch-1.10...
WARNING:root:TPU has started up successfully with version pytorch-1.10

[KrishPro]

I think this issue should be closed ?

[JackCaoG]

We still need to setup the nightly wheel for colab, will close after that.

[KrishPro]

torch_xla==1.9.0 and torch_xla==1.10.0 both are working quite well on Google Colab.

But recently, I tested on Kaggle Kernels.

torch_xla==1.9.0 is working as expected. But, torch_xla==1.10.0 doesn't seem to work. Btw, It is giving the same error ('GLIBC_2.28' not found').

!pip install --no-cache cloud-tpu-client==0.10 torch==1.10 https://storage.googleapis.com/tpu-pytorch/wheels/colab/1.10/torch_xla-1.10-cp37-cp37m-linux_x86_64.whl

Also doesn't work !

I have also tried multiple solutions, Including : -

!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py
!python pytorch-xla-env-setup.py --version 1.10 --apt-packages libomp5
import torch_xla

!pip install allennlp==2.9.1 fastai==2.5.3
!pip install tf-estimator-nightly==2.8.0.dev2021122109 earthengine-api==0.1.238 folium==0.2.1
!pip install torchtext==0.11.0 torchaudio==0.10.0 torchvision==0.11.1 torch==1.10.0
!pip install cloud-tpu-client==0.10 https://storage.googleapis.com/tpu-pytorch/wheels/tpuvm/torch_xla-1.10-cp37-cp37m-linux_x86_64.whl

!pip install allennlp==2.9.1 fastai==2.5.3
!pip install tf-estimator-nightly==2.8.0.dev2021122109 earthengine-api==0.1.238 folium==0.2.1
!pip install torchtext==0.11.0 torchaudio==0.10.0 torchvision==0.11.1 torch==1.10.0
!pip install cloud-tpu-client==0.10 https://storage.googleapis.com/tpu-pytorch/wheels/torch_xla-1.10-cp37-cp37m-linux_x86_64.whl

[KrishPro]

For Now, I am converting my code to work with torch==1.9.0.

Waiting for response from torch/xla developers, @yeounoh @JackCaoG

[Danc2050]

I also have this issue with 1.11.0 and 1.10.0.

[JackCaoG]

1.11 should work.. @yeounoh Could you take a look when you have time?

[kurianbenoy]

@KrishPro can you share the installation file for pytorch v1.9 which worked for you. Apparently pytorch/xla is not working in kaggle and colab

[Danc2050]

@kurianbenoy I used this (includes tensorboard):

!pip install cloud-tpu-client==0.10 torch==1.11.0 https://storage.googleapis.com/tpu-pytorch/wheels/torch_xla-1.11-cp37-cp37m-linux_x86_64.whl tensorboard-plugin-profile

[KrishPro]

$ curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py
$ python pytorch-xla-env-setup.py --version 1.9 --apt-packages libomp5 libopenblas-dev
$ pip install torchtext==0.10 torchaudio==0.9.0

I have manually changed the versions of torchtext and torchaudio. Because the env-setup.py changes (upgrade/downgrade) only the version of torch and torchvision to work with torch/xla. You can also confirm from the logs and the version of torchtext and torchaudio before and after running the script.

Note: PyTorch lightning was also having some issues the in-compatible torchtext and torch. This gave me hint to change the version of torchtext & torch

[KrishPro]

Are there any updates on this issue ? I am in a Kaggle Completion, I want to use label smoothing which is only present in 1.10+

I've currently created a custom cross entropy which offers label smoothing for 1.9. But still, We can't be on out-dated version of our main framework. Eagerly waiting for response from official developers.

@kurianbenoy I used this (includes tensorboard):

!pip install cloud-tpu-client==0.10 torch==1.11.0 https://storage.googleapis.com/tpu-pytorch/wheels/torch_xla-1.11-cp37-cp37m-linux_x86_64.whl tensorboard-plugin-profile

This doesn't work on kaggle either.

[yeounoh]

Hi @KrishPro , we moved the colab wheels to a different path. Could you try --force-reinstall using the following:

!pip install cloud-tpu-client==0.10 torch==1.11.0 https://storage.googleapis.com/tpu-pytorch/wheels/colab/torch_xla-1.11-cp37-cp37m-linux_x86_64.whl

cc @Danc2050

[Danc2050]

@yeounoh Works like a charm 🍀:

Collecting torch-xla==1.11
  Downloading https://storage.googleapis.com/tpu-pytorch/wheels/colab/torch_xla-1.11-cp37-cp37m-linux_x86_64.whl (152.9 MB)
Successfully installed cloud-tpu-client-0.10 google-api-python-client-1.8.0 torch-1.11.0 torch-xla-1.11
...
WARNING:root:Waiting for TPU to be start up with version pytorch-1.11...
WARNING:root:TPU has started up successfully with version pytorch-1.11

[KrishPro]

Hi @KrishPro , we moved the colab wheels to a different path. Could you try --force-reinstall using the following:

!pip install cloud-tpu-client==0.10 torch==1.11.0 https://storage.googleapis.com/tpu-pytorch/wheels/colab/torch_xla-1.11-cp37-cp37m-linux_x86_64.whl

cc @Danc2050

I am sorry for not being clear enough. But, torch/xla works great on colab for me too. I am having issues running it on kaggle.

Please let me know, if it works for you on kaggle and share me a working example

[Danc2050]

I am sorry for not being clear enough. But,
torch/xla works great on colab for me too. I am having issues running it on kaggle.

@KrishPro I believe the Kaggle notebooks use Google-backed spare/preemptible VMs, as do Google Colab machines, so it should be compatible.

[KrishPro]

I am sorry for not being clear enough. But,
torch/xla works great on colab for me too. I am having issues running it on kaggle.

@KrishPro I believe the Kaggle notebooks use Google-backed spare/preemptible VMs, as do Google Colab machines, so it should be compatible.

Can you make a public kaggle kernel and setup torch/xla on it and share it's link