Error: Rate Limit Exceeded

[starleaf1]

I encounter the following exception when I try to sync using the grive command

exception: /build/grive2-2Hxkkx/grive2-0.5.1/libgrive/src/protocol/AuthAgent.cc(159): Throw in function long int gr::AuthAgent::CheckHttpResponse(long int, const string&, const gr::http::Header&)
Dynamic exception type: boost::exception_detail::clone_impl<gr::http::Error>
[gr::expt::BacktraceTag*] = #0 0x578d79 grive gr::Exception::Exception()
#1 0x5597d7 grive gr::http::Error::Error()
#2 0x56119c grive gr::AuthAgent::CheckHttpResponse(long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, gr::http::Header const&)
#3 0x560c02 grive gr::AuthAgent::Request(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, gr::SeekStream*, gr::DataStream*, gr::http::Header const&, unsigned long long)
#4 0x560560 grive gr::http::Agent::Get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, gr::DataStream*, gr::http::Header const&, unsigned long long)
#5 0x556dd8 grive gr::v2::Feed2::GetNext(gr::http::Agent*)
#6 0x5442a9 grive gr::Drive::DetectChanges()
#7 0x50b9da grive Main(int, char**)
#8 0x50c1e9 grive main
#9 0x7fcead0c3830 /lib/x86_64-linux-gnu/libc.so.6 __libc_start_main
#10 0x50a4b9 grive _start

[gr::http::ResponseBodyTag*] = {
 "error": {
  "errors": [
   {
    "domain": "usageLimits",
    "reason": "rateLimitExceeded",
    "message": "Rate Limit Exceeded"
   }
  ],
  "code": 403,
  "message": "Rate Limit Exceeded"
 }
}

[skydivingpt]

+1 Ubuntu 16.04

Error :

Failed to obtain auth token: HTTP 400, body: { "error" : "invalid_grant" } exception: /build/grive2-2Hxkkx/grive2-0.5.1/libgrive/src/protocol/OAuth2.cc(80): Throw in function void gr::OAuth2::Auth(const string&) Dynamic exception type: boost::exception_detail::clone_implgr::OAuth2::AuthFailed [gr::expt::BacktraceTag*] = #0 0x578d79 grive gr::Exception::Exception() #1 0x5651a7 grive gr::OAuth2::AuthFailed::AuthFailed() #2 0x564271 grive gr::OAuth2::Auth(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) #3 0x50b576 grive Main(int, char**) #4 0x50c1e9 grive main #5 0x7f7d3ba2f830 /lib/x86_64-linux-gnu/libc.so.6 __libc_start_main #6 0x50a4b9 grive _start

[konradsa]

Same here:

exception: /home/pi/git/grive2/libgrive/src/protocol/AuthAgent.cc(159): Throw in function long int gr::AuthAgent::CheckHttpResponse(long int, const string&, const gr::http::Header&) Dynamic exception type: N5boost16exception_detail10clone_implIN2gr4http5ErrorEEE [PN2gr4expt12BacktraceTagE] = #0 0x14dabc :0 gr::Exception::Exception() #1 0x123380 :0 gr::http::Error::Error() #2 0x12b480 :0 gr::AuthAgent::CheckHttpResponse(long, std::string const&, gr::http::Header const&) #3 0x12b024 :0 gr::AuthAgent::Request(std::string const&, std::string const&, gr::SeekStream*, gr::DataStream*, gr::http::Header const&, unsigned long long) #4 0x121840 :0 gr::http::Agent::Get(std::string const&, gr::DataStream*, gr::http::Header const&, unsigned long long) #5 0x16dbe8 :0 gr::v2::Feed2::GetNext(gr::http::Agent*) #6 0x1099a8 :0 gr::Drive::DetectChanges() #7 0xe5b1c :0 Main(int, char**) #8 0xe62a4 :0 main #9 0x7684f294 /lib/arm-linux-gnueabihf/libc.so.6 __libc_start_main

[PN2gr4http15ResponseBodyTagE] = { "error": { "errors": [ { "domain": "usageLimits", "reason": "rateLimitExceeded", "message": "Rate Limit Exceeded" } ], "code": 403, "message": "Rate Limit Exceeded" } }

[gmyuriy]

Same here. At least can you make it fail gracefully and repeat failed request?

exception: /build/grive2-2Hxkkx/grive2-0.5.1/libgrive/src/protocol/AuthAgent.cc(159): Throw in function long int gr::AuthAgent::CheckHttpResponse(long int, const string&, const gr::http::Header&) Dynamic exception type: boost::exception_detail::clone_implgr::http::Error [gr::expt::BacktraceTag*] = #0 0x578d79 grive gr::Exception::Exception() #1 0x5597d7 grive gr::http::Error::Error() #2 0x56119c grive gr::AuthAgent::CheckHttpResponse(long, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, gr::http::Header const&) #3 0x560c02 grive gr::AuthAgent::Request(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, gr::SeekStream*, gr::DataStream*, gr::http::Header const&, unsigned long long) #4 0x560560 grive gr::http::Agent::Get(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, gr::DataStream*, gr::http::Header const&, unsigned long long) #5 0x556dd8 grive gr::v2::Feed2::GetNext(gr::http::Agent*) #6 0x5442a9 grive gr::Drive::DetectChanges() #7 0x50b9da grive Main(int, char**) #8 0x50c1e9 grive main #9 0x7fe401449830 /lib/x86_64-linux-gnu/libc.so.6 __libc_start_main #10 0x50a4b9 grive _start

[gr::http::ResponseBodyTag*] = { "error": { "errors": [ { "domain": "usageLimits", "reason": "rateLimitExceeded", "message": "Rate Limit Exceeded" } ], "code": 403, "message": "Rate Limit Exceeded" } }

[gr::http::ResponseCodeTag*] = 403 [gr::http::RequestHeadersTag*] = Authorization: Bearer GData-Version: 3.0

[gr::http::ResponseHeadersTag*] = HTTP/1.1 403 Forbidden Vary: X-Origin Content-Type: application/json; charset=UTF-8 Date: Wed, 16 Aug 2017 05:30:12 GMT Expires: Wed, 16 Aug 2017 05:30:12 GMT Cache-Control: private, max-age=0 X-Content-Type-Options: nosniff X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block Server: GSE Alt-Svc: quic=":443"; ma=2592000; v="39,38,37,35" Accept-Ranges: none Vary: Origin,Accept-Encoding Transfer-Encoding: chunked

[gr::http::UrlTag*] = https://www.googleapis.com/drive/v2/files?maxResults=1000&pageToken=***&q=trashed%3Dfalse

[bebo-dot-dev]

I see the same issue here occasionally. For me I believe it coincides with having performed some large maintenance task on my google drive i.e. deleting thousands of files / emptying the bin via the drive web app / drive android app. https://developers.google.com/drive/v3/web/handle-errors#403_rate_limit_exceeded

[kigiri]

Same here, I delete a somewhat big folder ~3000 files of 5mb each and now the sync fail with 403 Rate Limit Exceeded

[TopherGopher]

+1 I am having this issue as well. Here's a hypothesis (since I've seen this error writing my own GDrive playground stuff) - When you host an integration API app in your Google Drive account, there's a rate limit that you hit eventually. It's like - 3 billion requests, but it's still there - and with everyone using Grive now and the heavy number of files, could it be that we're exhausting Grive's Google integration's API limits? If the owner of the Grive integration went here, they could confirm (or smash) my theory. You can up the quota if so. Just a theory though.

[TopherGopher]

So I do have a workaround for folks who have a lot of files. TLDR; generate your own oauth creds and swap the default ones out, and then hack the max results in the list function to an insane number.

I changed these two lines of code and traded them out for creds I generated myself. These creds are actually free to generate and use. Go to Google developer console, enable the Drive API, create "other" oauth creds, and then just paste them in and re-build grive.

I added a hack to make less requests happen by changing maxResults=1000 to maxResults=999999999. This obviously consumes more memory, so tune as you need. For me, I have zillions of tiny files in flat directories, so this helped enormously.

---

I ended up using a simple Dockerfile to play around with building this (since my binary is currently installed from the apt repo, and I didn't want to grab some of the dev libs), and thought I'd pass it along. Ignore if it's worthless to you.

Save this as Dockerfile in the top directory of this repo

FROM ubuntu:16.04
RUN apt-get update && \
  apt-get install --yes git cmake build-essential \
  libgcrypt11-dev libyajl-dev libboost-all-dev \
  libcurl4-openssl-dev libexpat1-dev libcppunit-dev \
  binutils-dev pkg-config zlib1g-dev && \
  rm -rf /var/cache/apt/archives
VOLUME /data
WORKDIR /data

Build and run with: docker build -t grive2 . && docker run --name grive2 --rm -it -v $PWD:/data grive2:latest and then mkdir build && cd build && cmake .. && make -j4 to generate a new binary.

Because you're not sharing your requests with other people anymore, you aren't hitting the limits on the primary account. This also has the added benefit of you owning your own integration to your data.

[piekniewski]

@TopherGopher Cool, I did as you suggested - created the credentials, and recompiled grive2 but I seem to be getting unauthorized_client error. Any idea on what I might be doing wrong?

[piekniewski]

OK, figured that out. grive will not update auth token for new credentials for previously downloaded drive. One has to create a fresh directory, get a new auth token and download everything from scratch...

[TopherGopher]

Ahhh - glad you got it. Not the best solution, but I'm working on a cleaner one.

[alanmilinovic]

This is a show stopper for me. Getting this error all the time. Fix would be appreciated.

[cserpell]

This happens to me all the time. It is really annoying. I've got aroung 60 GB in Drive, and it takes a long time to sync, but it works around a third of the times I run it. Otherwise, it gives the following error (and maybe others sometimes).

When running grive -d:

HTTP GET "https://www.googleapis.com/drive/v2/files?maxResults=1000&pageToken=XXX&q=trashed%3Dfalse"
HTTP response 403
exception: /grive2/libgrive/src/protocol/AuthAgent.cc(159): Throw in function long int gr::AuthAgent::CheckHttpResponse(long int, const string&, const gr::http::Header&)
Dynamic exception type: boost::exception_detail::clone_impl<gr::http::Error>
[gr::expt::BacktraceTag*] = #0 0x562c46f09755 grive gr::Exception::Exception()
#1 0x562c46ee00c3 grive gr::http::Error::Error()
#2 0x562c46ee81dd grive gr::AuthAgent::CheckHttpResponse(long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, gr::http::Header const&)
#3 0x562c46ee7cf6 grive gr::AuthAgent::Request(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, gr::SeekStream*, gr::DataStream*, gr::http::Header const&, unsigned long long)
#4 0x562c46ede6ea grive gr::http::Agent::Get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, gr::DataStream*, gr::http::Header const&, unsigned long long)
#5 0x562c46f2a40c grive gr::v2::Feed2::GetNext(gr::http::Agent*)
#6 0x562c46ec9acd grive gr::Drive::DetectChanges()
#7 0x562c46eaa940 grive Main(int, char**)
#8 0x562c46eab142 grive main
#9 0x7fac15e9fb97 /lib/x86_64-linux-gnu/libc.so.6 __libc_start_main
#10 0x562c46ea935a grive _start

[gr::http::ResponseBodyTag*] = {
 "error": {
  "errors": [
   {
    "domain": "usageLimits",
    "reason": "rateLimitExceeded",
    "message": "Rate Limit Exceeded"
   }
  ],
  "code": 403,
  "message": "Rate Limit Exceeded"
 }
}

[gr::http::ResponseCodeTag*] = 403
[gr::http::RequestHeadersTag*] = Authorization: Bearer XXX
GData-Version: 3.0

[gr::http::ResponseHeadersTag*] = HTTP/1.1 403 Forbidden
Vary: X-Origin
Content-Type: application/json; charset=UTF-8
Date: Wed, 01 Aug 2018 18:20:08 GMT
Expires: Wed, 01 Aug 2018 18:20:08 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alt-Svc: quic=":443"; ma=2592000; v="44,43,39,35"
Accept-Ranges: none
Vary: Origin,Accept-Encoding
Transfer-Encoding: chunked


[gr::http::UrlTag*] = https://www.googleapis.com/drive/v2/files?maxResults=1000&pageToken=XXX&q=trashed%3Dfalse

[cserpell]

It seems @vitalif commited a fix 93cae255fa7a1af7f32a5271ab9e07964dbad389 that would help. Let's see what happens.

[edo0]

Any update on this issue? Keeps on presenting itself to me. $ grive Reading local directories Reading remote server file list request failed due to rate limiting: 403 (body: { "error": { "errors": [ { "domain": "usageLimits", "reason": "rateLimitExceeded", "message": "Rate Limit Exceeded" } ], "code": 403, "message": "Rate Limit Exceeded" } } ). retrying in 1 seconds request failed due to rate limiting: 403 (body: { "error": { "errors": [ { "domain": "usageLimits", "reason": "rateLimitExceeded", "message": "Rate Limit Exceeded" } ], "code": 403, "message": "Rate Limit Exceeded" } } ). retrying in 1 seconds request failed due to rate limiting: 403 (body: { "error": { "errors": [ { "domain": "usageLimits", "reason": "rateLimitExceeded", "message": "Rate Limit Exceeded" } ], "code": 403, "message": "Rate Limit Exceeded" } } ). retrying in 1 seconds request failed due to rate limiting: 403 (body: { "error": { "errors": [ { "domain": "usageLimits", "reason": "rateLimitExceeded", "message": "Rate Limit Exceeded" } ], "code": 403, "message": "Rate Limit Exceeded" } } ). retrying in 2 seconds request failed due to rate limiting: 403 (body: { "error": { "errors": [ { "domain": "usageLimits", "reason": "rateLimitExceeded", "message": "Rate Limit Exceeded" } ], "code": 403, "message": "Rate Limit Exceeded" } } ). retrying in 1 seconds request failed due to rate limiting: 403 (body: { "error": { "errors": [ { "domain": "usageLimits", "reason": "rateLimitExceeded", "message": "Rate Limit Exceeded" } ], "code": 403, "message": "Rate Limit Exceeded" } } ). retrying in 2 seconds request failed due to rate limiting: 403 (body: { "error": { "errors": [ { "domain": "usageLimits", "reason": "rateLimitExceeded", "message": "Rate Limit Exceeded" } ], "code": 403, "message": "Rate Limit Exceeded" } } ). retrying in 1 seconds request failed due to rate limiting: 403 (body: { "error": { "errors": [ { "domain": "usageLimits", "reason": "rateLimitExceeded", "message": "Rate Limit Exceeded" } ], "code": 403, "message": "Rate Limit Exceeded" } } ). retrying in 1 seconds request failed due to rate limiting: 403 (body: { "error": { "errors": [ { "domain": "usageLimits", "reason": "rateLimitExceeded", "message": "Rate Limit Exceeded" } ], "code": 403, "message": "Rate Limit Exceeded" } } ). retrying in 1 seconds request failed due to rate limiting: 403 (body: { "error": { "errors": [ { "domain": "usageLimits", "reason": "rateLimitExceeded", "message": "Rate Limit Exceeded" } ], "code": 403, "message": "Rate Limit Exceeded" } } ). retrying in 2 seconds request failed due to rate limiting: 403 (body: { "error": { "errors": [ { "domain": "usageLimits", "reason": "rateLimitExceeded", "message": "Rate Limit Exceeded" } ], "code": 403, "message": "Rate Limit Exceeded" } } ). retrying in 1 seconds request failed due to rate limiting: 403 (body: { "error": { "errors": [ { "domain": "usageLimits", "reason": "rateLimitExceeded", "message": "Rate Limit Exceeded" } ], "code": 403, "message": "Rate Limit Exceeded" } } ). retrying in 2 seconds request failed due to rate limiting: 403 (body: { "error": { "errors": [ { "domain": "usageLimits", "reason": "rateLimitExceeded", "message": "Rate Limit Exceeded" } ], "code": 403, "message": "Rate Limit Exceeded" } } ). retrying in 1 seconds request failed due to rate limiting: 403 (body: { "error": { "errors": [ { "domain": "usageLimits", "reason": "rateLimitExceeded", "message": "Rate Limit Exceeded" } ], "code": 403, "message": "Rate Limit Exceeded" } } ). retrying in 1 seconds request failed due to rate limiting: 403 (body: { "error": { "errors": [ { "domain": "usageLimits", "reason": "rateLimitExceeded", "message": "Rate Limit Exceeded" } ], "code": 403, "message": "Rate Limit Exceeded" } } ). retrying in 1 seconds request failed due to rate limiting: 403 (body: { "error": { "errors": [ { "domain": "usageLimits", "reason": "rateLimitExceeded", "message": "Rate Limit Exceeded" } ], "code": 403, "message": "Rate Limit Exceeded"

[jahagirdar]

Same issue here,

I am using my own key pair, When I run

grive -a --id "1020Something" --secret "DCSOMETHING"

I get

Reading local directories
Reading remote server file list
exception: /tmp/grive2/libgrive/src/protocol/AuthAgent.cc(164): Throw in function long int gr::AuthAgent::CheckHttpResponse(long int, const string&, const gr::http::Header&)
Dynamic exception type: boost::wrapexcept<gr::http::Error>
[gr::expt::BacktraceTag*] = #0 0x5602d5ef0897 grive gr::Exception::Exception()
#1 0x5602d5ec35b5 grive gr::http::Error::Error()
#2 0x5602d5ee1671 grive gr::AuthAgent::CheckHttpResponse(long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, gr::http::Header const&)
#3 0x5602d5ee0ea7 grive gr::AuthAgent::Request(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, gr::SeekStream*, gr::DataStream*, gr::http::Header const&, unsigned long long)
#4 0x5602d5ec19ee grive gr::http::Agent::Get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, gr::DataStream*, gr::http::Header const&, unsigned long long)
#5 0x5602d5f13412 grive gr::v2::Feed2::GetNext(gr::http::Agent*)
#6 0x5602d5ea776b grive gr::Drive::DetectChanges()
#7 0x5602d5e831d2 grive Main(int, char**)
#8 0x5602d5e83d7a grive main
#9 0x7f823a67a0b3 /lib/x86_64-linux-gnu/libc.so.6 __libc_start_main
#10 0x5602d5e814de grive _start

[gr::http::ResponseBodyTag*] = 
[gr::http::ResponseCodeTag*] = 403
[gr::http::RequestHeadersTag*] = Authorization: Bearer ya29SOMEThiNG
GData-Version: 3.0

[gr::http::ResponseHeadersTag*] = 
[gr::http::UrlTag*] = https://www.googleapis.com/drive/v2/files?maxResults=999999999&q=trashed%3dfalse

The URL on last list shows "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup."

Am I missing something?

[renedis]

same issue here.. Uploading 14.5TB and getting the error after 3.6TB on a 250/250 line.