How to check current BigQuery usage

[hugovk]

Whilst making https://github.com/ofek/pypinfo/pull/29 I got this:

    raise exceptions.from_http_response(response)
google.api_core.exceptions.Forbidden: 403 GET https://www.googleapis.com/bigquery/v2/projects/pypinfo-hugovk/queries/<snip>?maxResults=0&timeoutMs=10000: Quota exceeded: Your project exceeded quota for free query bytes scanned. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors

I can't find how much of my free monthly TB I've used up on the Google console.

After some digging in the console I found this:

https://console.cloud.google.com/home/activity?project=pypinfo-hugovk&authuser=1

Linking to this:

https://console.cloud.google.com/iam-admin/quotas?project=pypinfo-hugovk&authuser=1

Which isn't very informative -- all zeroes and dashes!

I get the error after doing several pypinfo --percent --pip pypinfo pyversion, but pypinfo --percent --pip -d 1 pypinfo is still fine.

Any idea what "QUERY-MBYTES-FOR-UNBILLED-PROJECTS-per-project" really means?

And where to check the monthly 1TB quota?

Thanks!

[ofek]

I'll look into this, thanks!

[hugovk]

Well, it carried on working a few hours later. I read somewhere (Stack Overflow?) that it's a rolling quota and will be replenished after some time. Still, it'd be nice to know where to check the actual quota at Google :)

I wonder, is it possible to show the amount of data a query makes, and possibly the remaining quota, when running?

[bskinn]

@hugovk I'm pretty sure there's a daily cap, in addition to the monthly quota. Every time I've gotten this error back from BigQuery, pypinfo has worked fine when I've tried it the next day.

No clue how to query the quota info, though. :-/

[hugovk]

That's also consistent with a rolling quota, meaning the quota applies to the month up to now. If you reach the limit, just wait a few hours and you'll get some more quota. No need to wait until the first (or whenever) of the month.

I've not found the quota page, but I have found one that shows how many bytes each query takes. I have it open on another computer and will post here next time I'm using it.

[hugovk]

Visit a page something like this, replace TODO_YOUR_PROJECT_HERE with your Pypinfo project name you chose in step 3 of the README:

https://console.cloud.google.com/home/activity?c=Configuration,Data_Access,Development,Monitoring,Platform,Recommendation&project=TODO_YOUR_PROJECT_HERE&authuser=1

On the filter on the right, make sure your project is selected in the dropdown, and enable all categories:

You should then see a bunch of logs like this:

The "Completed job" entries are the most interesting, showing the query pypinfo sent (you can see it locally with pypinfo --test), total billed bytes and total processed bytes:

From here, you can see that re-running a query that has been previously cached doesn't cost any extra bytes.

[ofek]

I added these stats just now in 9.0.0 :slightly_smiling_face:

Though, I still can't find a way to see one's remaining free quota.