Annotations missing from many panels

[Krinkle]

What happened:

An annotation that was succesfully saved with a tag, is only sporadically shown in panels on dashboards configured to show annotations with that tag.

How to reproduce it (as minimally and precisely as possible):

1. Go to https://play.grafana.org/d/000000016/1-time-series-graphs?orgId=1.
2. Alt-click somewhere in a timeseries panel on a dashboard.
3. Write "Hello" and tag it with "example".
4. Edit the dashboard config -> Annotations -> enable and query by tag "example".
5. Click "Apply", go back to dashboard, and click "Refresh dashboard" just in case.

Problem: The dashboard where the annotation was made, shows the annotation only in the panel where it was created. The other panels on the same dashboard do not show it!

Step 3	Step 4	Step 5: Problem

6. Go to a different dashboard.
7. Edit its config also to enable the same annotation, query by tag "example"
8. Click "Apply", go back to dashboard, and click "Refresh dashboard" just in case.

On most dashboards, it is now visible in all panels, which is great. But the dashboard where it originated shows it inconsistently. I also find that on some dashboards it is only showing on the first panel, even if it is not a repeating panel. I did not see anything obvious that is different from the dashboards where it works vs the ones where it shows only on the first.

Step 8: Problem

Anything else we need to know?:

I originally encountered this with Wikimedia's Grafana installation. The annotation is for 31 Aug 2022 14:00 and was created on the Traffic/Backend latency dashboard: https://grafana.wikimedia.org/d/O2sTrqZVk/backend-layer-performance?orgId=1&from=1661472000000&to=1662235200000&var-site=codfw&var-cluster=All

In the browser tab where I the annotation, no matter the number of refreshes only the first original panel shows it. No other panel repeats, row repeats, or other panels show it. In a logged-out browser tab, it seems to show for half of the panels but still not all:

For the Wikimedia install, when opening a different dashboard it does the annotation in all panels in all rows. This, unlike the above play.grafana.org example where even on a different dashboard it only showed in one panel.

https://grafana.wikimedia.org/d/000000066/resourceloader?from=1661472000000&to=1662235200000

Environment:

• Grafana version: 9.2.3-997144b7 (play.grafana.org), 8.5.14 (grafana.wikimedia.org)
• Data source type & version: Prometheus metric source, Grafana built-in annotation source.
• OS Grafana is installed on: Linux.
• User OS & Browser: Firefox latest, macOS.

[usmangt]

Hi @Krinkle,

Thanks for opening this issue and providing a nice very detailed description.

I am not quite sure about the answer so will ask our Engineering team so that they can have a look and can update on this issue.

[dprokop]

I'm not 100% about this either, but IIRC the annotations created on a given panel, are not shared between other panels on the dashboard.

@torkelo - Do you think this is a regression we have introduced with the Time series panel, or has it always been like that? Looking at old graph dashboards, the behavior is the same as described in this issue. Also, looking at the implementation I see the annotations are filtered per panel when merging panel and dashboard data - and this code has not been changed for quite a long time.

[torkelo]

@dprokop definitely looks like a regression

[torkelo]

@dprokop but when I tested this locally I could not replicate, but on play I could. Need to test more

[torkelo]

@dprokop @Krinkle I cannot replicate this on 9.3 https://play.grafana.org/d/000000010/annotations?orgId=1&from=1670578742382&to=1670579018681

the annotation added on the first panel with tag "deploy" is showing on all panels

[torkelo]

@Krinkle tested in on the dashboard you tried it on and it works for me

[grafanabot]

We've closed this issue since it needs more information and hasn't had any activity recently. We can re-open it after you you add more information. To avoid having your issue closed in the future, please read our CONTRIBUTING guidelines.

Happy graphing!

[helinmik]

We are having similar issues related to this. I think it happens when you use repeating rows/panels and annotate on those. It will not then annotate the other repeated panels (or panels in repeated rows), only in the original "spot".

I can create new ticket or test this issue if you want, but it seems to work that way on grafana play annotation as well as on our instance (Grafana v. 8.5.2.). So to repeat:

• go to https://play.grafana.org/d/000000010/annotations?
• Create a variable, e.g. custom 1,2,3
• add all option to the variable
• select the all option
• make panel be repeated based on that
• annotate on the 2nd repeated panel with some tag such as "deploy" in the play annotations
• annotation appears in the 2nd panel but not in the 1st or 3rd

[Phantuman]

I have run into the same problem, i tried to include annotations made on one dashboard into another one. here some of the annotations show up in one panel and some in another, but every annotation only in one of them. i zoomed out prior this, so there might be some caching involved.

I tried a little bit more. and the problem is that, every annotation is only shown in the panel with the same panelId as the one where it got created.

Here i added the panelId for better visualization.

[usmangt]

Hi @Phantuman

Which Grafana version you are using?

[Phantuman]

Grafana v10.1.0 (838218ba20)

[dabelenda]

I hit the same issue, when I tried annotations for the first time on Grafana v10.2.0 (migrated through multiple versions for a long time).

I created a new Dashboard (to rule out weird side effects due to dashboard migrations), created a panel, and made an annotation on that panel with a Tag.

On another dashboard I modified the default query to add the Tag to the query. The annotation would only display on the Pannel with the same Id as the one where I created the Annotation on the other Dashboard.

Then I thought that maybe the default Query "Annotations & Alerts (Built-in)" has some hard-coded behavior, and tried to create another annotation query using the Datasource "-- Grafana --" for the same Tag to be displayed on the Dashboard and This worked as expected.

So I think the issue here is that some hard-coded behavior for the default Annotations Query exists and the users are not aware of it.

[fatpat]

I hit the same issue, when I tried annotations for the first time on Grafana v10.2.0 (migrated through multiple versions for a long time).

I created a new Dashboard (to rule out weird side effects due to dashboard migrations), created a panel, and made an annotation on that panel with a Tag.

On another dashboard I modified the default query to add the Tag to the query. The annotation would only display on the Pannel with the same Id as the one where I created the Annotation on the other Dashboard.

Then I thought that maybe the default Query "Annotations & Alerts (Built-in)" has some hard-coded behavior, and tried to create another annotation query using the Datasource "-- Grafana --" for the same Tag to be displayed on the Dashboard and This worked as expected.

So I think the issue here is that some hard-coded behavior for the default Annotations Query exists and the users are not aware of it.

you got something I guess

I've been struggling for so long about using the builtin Annotations & Alerts and I can never make it to work in order to share one annotation to all panel of a dashboard.

Creating a new --Grafana-- annotations source almost works as expected. Previously created annotations now appear on all panels with the color configured in the new annotation. But when having 2 --grafana-- annotations made it impossible to add a new markers manually. When clicking in a panel, no more Add annotation button shows up :-1:

when changing the source of the builtin annotation to something else than --Grafana--, now it seems to work, I can add new annotations and there are shown in all panels of the dashboard.

I really think there is something about the builtin grafana annotations query, either it's a bug or a non obvious behavior

[jHumeLPL]

Yeah, having this exact issue as well.... Hopefully it gets fixed soon.

[marcelzwiers]

Same problem here (v10.3.1)

[marcelzwiers]

I had the problem described in this issue when I used the built-in query (named Dashboard below), but it all started to work as expected as soon as I created the same query (named Releases) next to the built-in one (which I then set to never show anything):

I couldn't find documentation what is so special about the built-in query and why you should or shouldn't use it

[nailgun]

I found how to fix it in Dashboard Settings -> JSON Model:

   "annotations": {
     "list": [
       {
         "builtIn": 1,
         "datasource": {
-          "type": "grafana",
-          "uid": "-- Grafana --"
+          "type": "datasource",
+          "uid": "grafana"
         },
         "enable": true,
         "hide": true,
         "iconColor": "rgba(0, 211, 255, 1)",
         "name": "Annotations & Alerts",
         "target": {
           "limit": 100,
           "matchAny": true,
           "tags": [
             "a"
           ],
           "type": "tags"
         },
-        "type": "dashboard"
+        "type": "tags"
       }
     ]
   },

Don't forget to change tag names!

Looks like a bug in the dashboard annotation settings editor.

[BinyanHu]

Same problem with v10.3.3

[lun-4]

I was able to reproduce this issue and the workaround described in https://github.com/grafana/grafana/issues/58275#issuecomment-2094110875 fixed it here, running grafana v10.4.2

[0lionelzhang0]

I found how to fix it in Dashboard Settings -> JSON Model:

   "annotations": {
     "list": [
       {
         "builtIn": 1,
         "datasource": {
-          "type": "grafana",
-          "uid": "-- Grafana --"
+          "type": "datasource",
+          "uid": "grafana"
         },
         "enable": true,
         "hide": true,
         "iconColor": "rgba(0, 211, 255, 1)",
         "name": "Annotations & Alerts",
         "target": {
           "limit": 100,
           "matchAny": true,
           "tags": [
             "a"
           ],
           "type": "tags"
         },
-        "type": "dashboard"
+        "type": "tags"
       }
     ]
   },

Don't forget to change tag names!

Looks like a bug in the dashboard annotation settings editor.

This is still bugged in Grafana v11.1.0 (5b85c4c2fc) Modifying the json as suggested above works for the built-in annotation query.