Metrics Refactoring udata 2.1.0 No statistics gathered

[mayjolux]

Hey nice to see the new udata releases! We are still working hard here in Luxembourg to finish the udata 2 migration.

Now to my question, I am still migrating our prod from udata 1.6.20 to udata 2.0.0. But recently we noticed that on udata 1.6.20 Piwik did not longer track the site despite having all the latest udata packages for piwik and its necessary config changes, so I made a rollback to udata 1.6.19 which solved our problem immediately.

Since we don't want to stay any longer on a pre-udata2 setup, I worked my way around to udata 2.1.0 with the metrics breaking changes. So I began modifying our setup in such a way that we have a working influxdb instance with the correct port open, made sure that the udata-metrics package is installed with the right settings like database name etc. and tested the setup by pinging the influxdb to see if the connection works.

In that matter everything seems to work, but if I run any udata piwik or metrics command, I receive no error message and everything runs just fine, but there is no data added to the influxdb and therefore also no data pushed to piwik. I am currently a little lost in this influxdb matter, since I don't know if I have to do additional things concerning this, like creating tables or something else perhaps? Since the documentation of udata has become very scarce, I have to gather all the information by myself from all the bits I can find, therefore I am asking you once again for help in this matter.

Thank you for any help you can give me!

[abulte]

Hi,

There should be no data being sent from Influx to Piwik.

The process is:

• data is sent to Piwik, either directly through the javascript tracking code from the browser, or from udata's celery task piwik_track_api
• data is pulled from Piwik to Influx (in order to maintain a daily counter foreach model object) via a scheduled celery task
• data is pulled from Influx into udata's object model (in order to maintain a global counter foreach model object) via another celery task

Given your problem description, I'd focus on the first part of the process. You should have data in Piwik no matter what. Then you can move on to synchronising data from Piwik to udata via Influx. Maybe something is wrong w/ your udata-piwik install or conf? Do you see the tracking requests going out from the browser to Piwik for example?

BTW, you should go through 1.6.20 when migrating to udata 2.0.0 because of https://github.com/opendatateam/udata/pull/2305: it will ensure your queued celery tasks are compatible w/ udata 2 new serialisation format.

[mayjolux]

Thanks I will have a look at the tracking requests, but as mentioned I did a Rollback from 1.6.20 to 1.6.19 on our production side because we noticed that Piwik stopped working on that version, despite having all the correct settings. I suppose it should work on either version, because nothing major should change.

Version: udata-piwik==1.5.1 and of course I added the fix for the breaking change and added the PIWIK_ID_FRONT and PIWIK_ID_API to the config, but yet if I run our System on 1.6.20 the tracking stops working while working fine on 1.6.19.

Looking at our staging side which is currently operating at udata 2.1.3 and if we look there for data on our piwik, nothing is added anymore, therefore I assumed it would have something to do with the new influxdb. But there seems to be a breaking change since 1.6.20, which I can currently not identify.

[abulte]

udata==1.6.20 + udata-piwik==1.5.1 should indeed work, this is what we had in production at the some point. I don't see anything in the changes from 1.6.19 to 1.6.20 that could break Piwik, except maybe the celery tasks refactoring but you should at least have some data in Piwik from the browser.

Like I said you can check that the tracking requests are made, and if they're not that the tracking code is included in the HTML. We can continue to troubleshoot from there ;-)

[mayjolux]

At the end it was a piwik problem that got resolved rather quickly. The data is now being tracked again and then put into the influxdb. We finally migrated our prod to udata2 yesterday (yeah!) and it seems that i still have some open questions for you about the metrics. We freshly deployed new machines with udata2, I restored our data, but it seems that the countertags of the downloads are different than from before(several hundreds in difference), due to our covid-19 reports being published on a daily basis we have a high count of people visiting those datasets, but still the counters barely move anymore. I can be wrong, but before the migration I had the jobs piwik-current-metrics, piwik-yesterday-metrics and count-tags cronned, now it seems that the counters are not updating anymore on our site. Do you have any suggestion where to look first, I see that there is now also a piwik-update-metrics, but not much is happening when I run this manually.

[abulte]

Great news for the migration, congrats!

About downloads counters: it's working a lot better for us in production since we made the influx/refactoring changes, so there's hope for you.

Related to piwik, this is the jobs we currently have scheduled:

udata@host:~$ udata job scheduled|grep piwik
Job piwik-bulk-track-api(800): piwik-bulk-track-api(800) ↦ * * * * *
Job piwik-current-metrics: piwik-current-metrics ↦ 0 */6 * * *
Job piwik-update-metrics: piwik-update-metrics ↦ 59 */6 * * *

piwik-bulk-track-api will send the API calls to Piwik when 800 of them have been stored internally. This scheduled job is not needed if PIWIK_BULK == False in settings. In this case the API calls will be sent "live".

piwik-current-metrics will get the current day's data (downloads...) from Piwik and send them to influx via udata-metrics for aggregation.

piwik-update-metrics will get from Influx the modified objects since 24h, and for those objects the aggregated metrics counts, and write those in the udata's model. This is the values that will be displayed on the site.

Hope this helps.

Pinging @quaxsze here because I'm on holidays next week and he wrote most of the metrics refactoring so he should be able to help :-)