utm_campaign and campaign seem to not work

[rouilj]

I work on software that is meant to be self-hosted. It includes a link to its documentation site.

I would like to find out when somebody hits the documentation site from an instance of the software. Currently I get the referer and from that I can kind of tell when the referer is an instance of the software. However I would like to change the link to a no-referer link so I don't get internal DNS names or IP addresses/ports.

As a replacement for the referer, I would like to use campaign to tell me when the source is a self hosted Roundup instance. However it doesn't seem to work. The get to GoatCounter sends:

utm_campaign: from_roundup-2.3
ref: from_roundup-2.3.0

I have used campaign in place of utm_campaign but nothing shows up in the campaign widget and running

$ ./goatcounter db query 'select hit_id, ref, campaign from hits order by hit_id desc limit 5;'
hit_id  ref                                         campaign
53285   from_roundup-2.3.0                          NULL
53284   from_roundup-2.3.0                          NULL
53283   www.roundup-tracker.org/docs/FAQ.html       NULL
53282   www.roundup-tracker.org/docs/overview.html  NULL
53281   from_roundup-2.3.0                          NULL

doesn't show my campaign string.

This is with a self compiled: Goatcounter version: 23a43ea5_2023-09-03T14:22:33+02:00

What am I missing?

Thanks.

[patricksalo]

I can confirm that using campaign and source initially don't appear to work. Last month I started using GoatCounter to track a campaign and spent several hours trying to get it to work and just like rouili, nothing would show up in the Campaigns widget in the dashboard. At some point things magically started working.

Now move forward to today. Yesterday I was updating my prior campaign with new parameters for a new campaign, and yet again, I couldn't get this new campaign to show up in the dashboard. I tried switching over to utm_source, tried a campaign with no source, etc. I updated GC from 2.4.1 to 2.50, tried manually invoking the Settings » Background tasks, tried all kinds of different browsers, etc. All to no avail.

Here I am the next day and 12 hours later, and lo and behold, GC now shows this new campaign in the dashboard including all the hits that occurred yesterday when I was trying to get things to show up. Also now that this new campaign is visible in the dashboard, if I hit one of my new campaign URLs, I can immediately see the hit in the dashboard (within a few seconds to minute). @arp242 My guess is there's some kind of display bug here that is time based.

Upon a bit more sleuthing this morning, I've also discovered that all of my hits are only showing up for today and I'm seeing zero for yesterday (e.g. all of last night's hits are attributed to today's date, not yesterday). Coincidentally I was also testing this out yesterday from 18:00 - 22:00 my time (Pacific) and within my GC Profile I have this time zone selected. Is GC maybe using UTC for all time calculations and hence all date selections in the Dashboard aren't based upon your local timezone? This might only be a piece of the puzzle though (or maybe an entirely different issue?) as last month when I started down the campaign path, I was doing all my work on the other end of the day, mornings from about 09:00 - 11:00 (Pacific).

Let me know if there's any logs or troubleshooting I can provide. Also if it helps, I'm using PostgreSQL for my backend db.

P.S. One more thing to add to help root cause this. I did notice last night that I couldn't get my current (last month's) campaign to show any new hits or increment in any way. So in essence when I departed my GC sleuthing, I couldn't get GC to display any campaign hits, whether for the current campaign or the new one I was adding. Also just like with my new campaign, this morning I'm now seeing hits I was doing yesterday to the old/current campaign. So there's definitely some indication that maybe GC hits a point somehow, where it's capturing hits, just not displaying them until later whether it's a wait until the day rolls over or a wait X# of hours and things somehow get rectified.

[arp242]

If I send e.g. (spaces added for readability):

/count ? p=/ & q=utm_campaign=from_roundup-2.3

Then this shows up in the dashboard.

Although it will only be processed if the same person hasn't viewed that path in the last 8 hours. So if someone has visited http://docs.roundcube.com and then later clicks that campaign link, nothing will show up. Also visiting /path1?campaign=XXX and then /path2?campaign=XXX will make it show up twice.

Arguably that makes sense? idk. Maybe it should keep track of unique campaigns separately from paths 🤔

Not sure if this is the problem here, but probably.

[arp242]

Did fix storing the hits.campaign column with 4353352 by the way. But note that the hits table is never queries, and hasn't been since 2020 or thereabouts. It exists purely for CSV exports and maybe some debugging purposes. The correct table is the aggregate campaign_stats.

[patricksalo]

@arp242 Thanks for the insights. What's odd though is that from what I can tell the hits themselves are captured, but are not displayed on the dashboard until a day later.

[arp242]

I can't think of why that would happen; updating the campaign stats is all in the same code as everything else (browsers, refs, actual pageviews, etc.)

[arp242]

One possibility might be timezones; most the horizontal bar charts (browsers, systems, campaigns) are collected by day in UTC. The reasoning here is that you don't really care if there were 60 or 70 people with Firefox today, but generally you care about proportions (e.g. 20% Firefox, 30% Chrome, etc.) and it's a lot more efficient to do it like this.

For pageviews and refs it stores them by hour instead of day, which is adjusted for your local TZ when viewing. We can't do this for anything else as we don't have that information.

For campaigns that might not be the correct, because here you care more how many per actual day.

What is your local TZ? I think this might be your issue?

[patricksalo]

What is your local TZ? I think this might be your issue?

PDT/PST over here. I'm off -7 or -8 from GMT depending upon what time of year it is.

P.S. I was thinking the same thing with this potentially being a time zone issue.