HTTP Error 403: rate limit exceeded when loading model

[ichamatidis]

🐛 Bug

To Reproduce

Steps to reproduce the behavior:

1. import torch
2. model = torch.hub.load('pytorch/vision:v1.9.0','resnet50', pretrained=True)

Expected behavior

Load the model

Environment

Google Colab and also local with the latest pytorch and torchvision installed

Error Output

HTTPError Traceback (most recent call last)

in () 1 import torchvision 2 ----> 3 model = torch.hub.load('pytorch/vision:v1.9.0','resnet50', pretrained=True)

8 frames

/usr/lib/python3.7/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs) 647 class HTTPDefaultErrorHandler(BaseHandler): 648 def http_error_default(self, req, fp, code, msg, hdrs): --> 649 raise HTTPError(req.full_url, code, msg, hdrs, fp) 650 651 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 403: rate limit exceeded

[oke-aditya]

I'm able to download on my local machine. With the following command

import torch
model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet50', pretrained=True)

Also with

import torch
model = torch.hub.load('pytorch/vision:v0.9.0', 'resnet50', pretrained=True)

Can you give one more try @ichamatidis ?

[ichamatidis]

I tried it with both versions that you mentioned, still the same result in Google Colab and locally. I provide bellow the error output form the 0.10.0 version from Google Colab.

Error Output

---

HTTPError Traceback (most recent call last)

in () 1 import torch 2 ----> 3 model = torch.hub.load('pytorch/vision:v0.10.0','resnet50', pretrained=True)

8 frames

/usr/lib/python3.7/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs) 647 class HTTPDefaultErrorHandler(BaseHandler): 648 def http_error_default(self, req, fp, code, msg, hdrs): --> 649 raise HTTPError(req.full_url, code, msg, hdrs, fp) 650 651 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 403: rate limit exceeded

[oke-aditya]

True. I can reproduce this over Google Colab. Not locally though

Here is link

Unsure why colab should do that.

But you can access pretrained model from rnet = torchvision.models.resnet50(pretrained=True)

[ichamatidis]

Yeah it's a weird bug and must be recent(less than 2 weeks cause it was working normally before). Does your local machine has Windows or Linux?

[oke-aditya]

Linux Ubuntu 20.04.2

[ichamatidis]

I have Ubuntu 21.04, so I assume it's some connection error. Thank you for your advise :)

[ptrblck]

Issue is also observed here and I'm also able to reproduce it using Ubuntu20.04.

[pmeier]

This seems to be a bug in torch.hub with pytorch/pytorch#56138 as possible offender. I can't reproduce the issue with torch==1.8.0 which does not contain this commit. cc @cyyever @ailzhang

[ailzhang]

Yea I think it's introduced in that PR cc: @cyyever. It's probably hitting the rate limit set by github. My guess is it's only reproducible on CI machines/Google Colab since those machines might send a lot of API requests from the same public IP address so github only errors out on those. Our personal dev machines didn't hit the limit so it's not reproducible there. If we still don't want to introduce dependency on system git, we probably need to either find another way to validate branch/tag or somehow workaround the rate limit.

[AK391]

torch==1.8.0

can confirm using torch==1.8.0 and restarting runtime in colab works

[Vernerrf]

Hi everyone, With me the error happens with any model used. And even resetting the collab the problem persists.

[pmeier]

@ailzhang I can reproduce this locally and @NicolasHug seems to have hit the same in #4157. So it is not only CI machines and Colab.

---

Bumping priority due to user activity.

[SomTambe]

I'm getting the same error even while trying to run the torch hub models on docker containers. In my opinion, that should not happen. The system is running Ubuntu 20.04.

[cyyever]

This is because we are using GitHub API to check the existence of the repo. I have submitted a PR to ignore HTTP error 403. To work around this error temporarily, you can delete line 163 on torch/hub.py

_validate_not_a_forked_repo(repo_owner, repo_name, branch)  

[pmeier]

pytorch/pytorch#61761 for reference.

[NicolasHug]

Also @ichamatidis :

torch.hub.load('pytorch/vision:v1.9.0','resnet50', pretrained=True)

This should be either vision:v0.9.0 or vision:v0.10.0, as torchvision's version 1.9 doesn't exist.

[ichamatidis]

@NicolasHug Yeah copy-paste error, but the HTTP error persists even with wrong parameters, thus we can narrow that the error happens before any checks

[NicoHambauer]

Similar error here!

[deepsworld]

Noticed similar issue on personal machine with pytorch:1.9 and torchvision:0.10.0

[farleylai]

Likely something to do with pytorch=1.9:

• Works with python=3.7 and pytorch=1.7

Python 3.7.10 | packaged by conda-forge | (default, Feb 19 2021, 16:07:37) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.7.0'
>>> model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet50', pretrained=True, force_reload=True)
Downloading: "https://github.com/pytorch/vision/archive/v0.10.0.zip" to /home/ml/farleylai/.cache/torch/hub/v0.10.0.zip

• Works with python3.9 and pytorch=1.7.1 and 1.8.1.

Python 3.9.5 (default, Jun  4 2021, 12:28:51) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
import torch
>>> torch.__version__
'1.7.1'
>>> model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet50', pretrained=True, force_reload=True)
Downloading: "https://github.com/pytorch/vision/archive/v0.10.0.zip" to /home/ml/farleylai/.cache/torch/hub/v0.10.0.zip

Python 3.9.5 (default, Jun  4 2021, 12:28:51) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.8.1'
>>> model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet50', pretrained=True, force_reload=True)
Downloading: "https://github.com/pytorch/vision/archive/v0.10.0.zip" to /home/ml/farleylai/.cache/torch/hub/v0.10.0.zip

• Failed with python=3.9 and pytorch=1.9

Python 3.9.5 (default, Jun  4 2021, 12:28:51) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.9.0'
>>> model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet50', pretrained=True, force_reload=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ml/farleylai/miniconda3/envs/sinet39/lib/python3.9/site-packages/torch/hub.py", line 362, in load
    repo_or_dir = _get_cache_or_reload(repo_or_dir, force_reload, verbose)
  File "/home/ml/farleylai/miniconda3/envs/sinet39/lib/python3.9/site-packages/torch/hub.py", line 162, in _get_cache_or_reload
    _validate_not_a_forked_repo(repo_owner, repo_name, branch)
  File "/home/ml/farleylai/miniconda3/envs/sinet39/lib/python3.9/site-packages/torch/hub.py", line 124, in _validate_not_a_forked_repo
    with urlopen(url) as r:
  File "/home/ml/farleylai/miniconda3/envs/sinet39/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/home/ml/farleylai/miniconda3/envs/sinet39/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/home/ml/farleylai/miniconda3/envs/sinet39/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/home/ml/farleylai/miniconda3/envs/sinet39/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/home/ml/farleylai/miniconda3/envs/sinet39/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/home/ml/farleylai/miniconda3/envs/sinet39/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: rate limit exceeded

[mattpopovich]

Similar as @farleylai, no dice with pytorch 1.9.0. This was working previously with pytorch 1.7.0 and 1.8.0. Although, that could be a coincidence if network issues are the problem.

root@pc:~#  python3
Python 3.8.10 (default, Jun  2 2021, 10:49:15) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.9.0a0+gitd69c22d'
>>> model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet50', pretrained=True, force_reload=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/torch/hub.py", line 362, in load
    repo_or_dir = _get_cache_or_reload(repo_or_dir, force_reload, verbose)
  File "/usr/local/lib/python3.8/dist-packages/torch/hub.py", line 162, in _get_cache_or_reload
    _validate_not_a_forked_repo(repo_owner, repo_name, branch)
  File "/usr/local/lib/python3.8/dist-packages/torch/hub.py", line 124, in _validate_not_a_forked_repo
    with urlopen(url) as r:
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: rate limit exceeded

[pmeier]

No need for more repros. This behavior was introduced in pytorch/pytorch#56138 which was shipped with 1.9.0 and will be fixed by pytorch/pytorch#61761. I don't know if we release a 1.9.1 that includes this fix since it seems to affect quite a few users (@seemethere ?). If not we have to wait for the 1.10 release.

[shankhanil007]

I have a temporary fix for this problem.

" base_model= torch.hub.load('pytorch/vision:v0.9.0', 'densenet121', pretrained=True, progress=True) " : gives me "HTTP Error 403: rate limit exceeded"

BUT,

" base_model = torchvision.models.densenet121(pretrained=True, progress=True) " : works completely fine

[pmeier]

Yes, this is a problem with torch.hub. You can access everything through torchvision directly.

[oke-aditya]

Yes the hack fix for now is to use

model = torchvision.models.densenet121(pretrained=True, progress=True)

instead of

model = torch.hub.load('pytorch/vision:v0.9.0', 'densenet121', pretrained=True, progress=True)

Note that this hack has a small problem. torch.hub does not need torchvision to be installed. It will load the model without torchvision being installed through pip / conda. Whereas the hack to load using torchvision needs torchvision to be installed.

[pmeier]

the hack fix

This is not a hack, but a stable feature of torchvision. If you have torchvision installed, one could also say it is superior, because torch.hub downloads the repo in addition to the weights. Still, you are correct, torch.hub is the way to go if you don't have access to torchvision or don't want it as dependency.

For anyone wondering what can be loaded by torch.hub, have a look at

https://github.com/pytorch/vision/blob/300a8a40132c5d3ee7dc936034c6046fd30ca4c0/hubconf.py#L1-L21

[mattpopovich]

Temporary fix by @malfet that he mentioned in this comment that fixes things for me:

As a workaround, can you please adding the following line before making any "torch.hub" calls:

torch.hub._validate_not_a_forked_repo=lambda a,b,c: True

[dywsjtu]

Temporary fix by @malfet that he mentioned in this comment that fixes things for me:

As a workaround, can you please adding the following line before making any "torch.hub" calls:

torch.hub._validate_not_a_forked_repo=lambda a,b,c: True

That works for me too.

[wally456456]

You can simply download pretrained model by torchvision.model
it seems like there is bug in torch.hub

[monajalal]

I was able to reproduce the error on CentOS 7 https://stackoverflow.com/questions/68901236/urllib-error-httperror-http-error-403-rate-limit-exceeded-when-loading-resnet1