Allow multiple OAuth Credentials

[Starker3]

With the upcoming sunset date for GA fast approaching, we've had quite a few users who've reached out to us asking if there is a faster way to import their GA data for all of the sites that they are trying to import.

The main limiting factor in regards to the speed at which the import can be done is the 10,000 daily API requests limit imposed by Google.

One potential solution around the per-project API limit would be for users to create a new OAuth Credential and Project for each different site that they want to import into Matomo. This would allow each site to have its own 10,000 daily API limit and not negatively impact all the other imports that are already running.

However, this wouldn't be possible with the current setup of the plugin as only one OAuth Credential can be uploaded at a time and only one token can be stored in the Matomo database for the GA importer. That means that if they created new projects as explained above, they would effectively be stopping their already running imports each time they uploaded a new OAuth Credential and went through the Auth steps for that new OAuth setup.

[jakedowns]

i think this is a unique workaround approach.

i thought about trying to request a quota increase through google, but who know's what kind of a snowball's chance i'll have with that route.

[gfaugere-safti]

From my testing, the per-site 10,000 limit on Google side is strict - however you can reach it on multiple sites with the same credentials.
On matomo's side, a flag is set when this limit is reached, preventing ANY site from continuing even if these did not reach the limit yet
By crudely running the SQL query DELETE FROM matomo_option WHERE option_name = 'GoogleAnalyticsImporter_nextAvailableAt'; before launching imports through the CLI, I can successfully run multiple imports and go as high as 10,000 query/site/day

[AltamashShaikh]

@gfaugere-safti This is done when the API sends a 403/429 and errorMessage has the word "daily" meaning the daily limit has reached and we backOff for that day.

Reading your comment you are trying to say that this limit is for each site right ? If you run the importer for a different site by deleting the option from db it works ?

[gfaugere-safti]

Yes exactly

There are multiple limits:

• a 10,000 requests/site/day, which triggers this backOff on the plugin side, which I think should not (or should back off only for this site)
• a 50,000 requests/day global limit, which applies to all

I'm not sure if there is a way to detect which limit was reached when receiving an error

[AltamashShaikh]

@gfaugere-safti Thanks for confirming, will check with our product team to get this task prioritised.

[AltamashShaikh]

@gfaugere-safti A new version with the desired change should be live soon.

[gfaugere-safti]

Well done, thanks for your work :)