Tracking API: provide a new JSON Tracker API

[mattab]

Our tracking API is fully documented at: Tracking API - tracking users analytics

It is also fully featured, and yet still a bit unstructured. Because we have added parameters one after the other when they were needed for useful features, now we realize that they could be named better, and be more structured.

Now that JSON has gone mainstream, it is natural to provide a pure JSON proxy to our good old named parameters. In the making, we could introduce structure in elements like events or ecommerce interactions (define each as a JSON object).

New API names proposals (to be discussed)

• rec -> record
• idsite -> idSite
• url -> actionUrl
• action_name -> actionName
• gt_ms -> actionGenerationTimeMs
• _id -> visitId
• _cvar -> pageCustomVariables
• urlref -> urlReferrer
• _idvc -> visitCount
• _viewts -> tsPreviousVisit
• _idts -> tsFirstVisit
• _rcn -> campaignName
• _rck -> campaignKeyword
• res -> resolution
• h+m+s -> merge into userLocalTime
• ua -> userAgent
• lang -> userLanguage?
• cvar -> pageCustomVariables
• search -> searchKeyword
• search_cat -> searchCategory
• search_count -> searchResultsCount
• idgoal -> idGoal
• group all ec_* into "ecommerce" object
• group all location info (country, region, city, latitude/longitude) into "location" object
• group all events e_* into an "event" object

[mattab]

See also #6000 Build a RESTful API for Piwik

[Spone]

Hi @mattab ,

I've landed there from piwik/developer-documentation#28

I think you should consider using POST request for the Tracking API. Maybe you can also allow GET requests in case the client cannot do POST requests (Google Analytics does that), but the better HTTP verb to use in this case in POST (see here).

Some other thoughts:

apiv will now be passed in the URL (for instance http://mypiwik.com/api/v2/) if I understand correctly.

lang -> userLanguage?

I would name it locale or userLocale, and allow just ISO 639 (language) code (optionally with an ISO 3166-1 alpha-2 (2-letter country) code).

Also, I don't know if you're familiar with JSON Schema, but it may be a good way to validate such user-submitted data.

Best, Hans

[fulstadev]

Hi @mattab,

Is this feature planned anytime soon? It is an issue when trying to integrate Matomo as a 'destination' in customer data platforms, particularly when it comes to authentication, which is required to forward the correct timestamp and IP of an event in such cases. For safety reasons, it's better to forward the token_auth only within the POST request body, as your docs also say. Yet this is unfortunately not possible with tools that only allow to forward data with the application/json content-type, as long as Matomo's Tracking API only understands x-www-form-urlencoded POST requests.

It would also be more user-friendly to build variable request payloads, instead of needing to worry about concatenating all query parameters followed by URL-Encoding. A great plus for people who cannot code and construct their requests within a CDP, filling the according key-value pairs into text fields. That part of this comment is a detail, of course, but making it easier for no-devs to use your tracking API, why not (just a thought).

But the problem with the authentication described above technically forces us to make a roundtrip to our own server just because of the Content-Type and the token_auth, except we're missing something? The only other solution we see is exposing to token_auth as query parameter in the HTTP requests that forward events received in our CDP to Matomo?

[mattab]

@fulstadev Thanks for your feedback! I was wondering about this: people who cannot code and construct their requests within a CDP, filling the according key-value pairs into text fields. It's a detail, of course, but it would it easier for no-devs to use your tracking API, too. -> Could you maybe share some screenshots showing, what the experience is like for you, when using your CPD and trying to construct the request? What tool do you use?

[fulstadev]

@mattab Sure, no problem. If the Matomo API could understand JSON payloads, you could use the default masks these tools often offer, in a simple way (the 'webhook URL' would be the URL representing a matomo tracking API call, and what you see in the image with the '$'-prefix are (JSON) variables from the event incoming into the CDP.

But because Matomo only understands the parameters via URL, you instead have to prepare the URL using sth like:

This is fine as long as you don't have any spaces or whatever, which is when URL-encoding is needed. The tool we use actually does not offer a URL-encoder for the generation of the webhook URL to be used as data pipeline destination. Consequently, the only solution we see is to generate the URL (url-encoded) to be used for the matomo tracking API call entirely within our backend, and send that URL along as attribute of any event we send to our CDP. Then we can use the ready URL within the CDP, and create an automated pipeline. But note that this still does not solve the issue of the token_auth, as the HTTP request the tool sends is an application/json request (which by the way also eliminates any URL-encoding issues anyway). So either we still send the token_auth as GET parameter, or we cannot set the timestamp and the IP of a tracked event, which means that the timestamp and IP of every event that is forwarded from our CDP into Matomo are actually incorrect (for example categorizing the incoming event to originate from the country of the infrastructure of the CDP, we could already confirm that when weird sequences of country-A <-> country-B showed up in our events history).

Another thing is that Matomo Tracking API URLs quickly become long. If no-devs have to use sth like 20 placeholders within it, join them with ampersands, and then watch out that everything is url-encoded (if the code permits it), it can be tedious to build this with pure placeholders, again for non-devs. It also happened to our devs already that a pretty long Matomo tracking URL failed to be executed because an ampersand was missing in front of one of the placeholders, and the resulting matomo request was treated as a 400 and broke the flow. Stupid stuff like that may occur more easily, that's the experience we can share.

So we're not sure how you weigh these issues, but the token issue is of greater importance for us..? Thanks anyways for your quick reaction!

And please don't misunderstand this post, we really appreciate matomo and its features, a lot.

A last note regarding the UX with CDPs though is that we've been working with more than three of the rather big players, and they were not offering an out-of-the-box integration with matomo, as with other analytics tools. Hence the only workaround for a data pipeline through to matomo is an actual webhook endpoint as data pipeline target (probably already at the dev-level here). We just think that a JSON-version of your API would make that integration much easier.

[mattab]

Thanks for the feedback @fulstadev, that's appreciated!