Backup, restore, and delete teams, team-members, and library-elements

[nileger]

Problem

It was not possible to backup and restore any library-elements (library-panels) and teams. However, we heavily rely on library-panels because they are referred on multiple dashboards. Without library-elements being backed up, all our dashboard are broken after a restore.

Solution

This PR solves the above described problem by extending the grafana-backup-tool to also back up library-elements and teams.

Testing

Basic tests were already successful. I'm currently doing more advanced tests. Any feedback and and testing is highly appreciated.

Related Issues

#180 #164

[acjohnson]

+1 this looks good to me, any feedback @ysde or others?

[nileger]

@acjohnson I just found out that when restoring, references between teams and users as well as dashboards and library panels are lost. Thus, I have to have to take a look at it tomorrow.

[nileger]

@acjohnson Backing up library-elements, teams, and team members works now. Unfortunately, restoring library elements didn't work in v8.0.0 for me. This is because Grafana creates the library-elements with the wrong uid. However, this is fixed in v8.4.3.

[ysde]

Hi @acjohnson Sorry for the late response

Hi @NilsLeger Thanks for the PR.

Is it compatible for older version grafana ? I mean if older version of grafana doesn't support library-element api, would this PR break the backup or save process ?

Thank you

[nileger]

Hi @ysde, thank you for your input.

I'm not sure whether backups or restores for older Grafana versions would stop working with this update. Irrespective of that, we should implement functionality to skip specific types of resources for specific version. Then, we are 100% sure that this PR is no breaking change.

I thought of something like the following procedure.

1. The health endpoint returns the Grafana version.
2. We have a map that maps resource types to Grafana versions (e.g., library-elements: 8.4.0)
3. Based on 1. and 2., we decide whether to backup and restore a certain resource type.

What do you think?

[ysde]

Hi @ysde, thank you for your input.

I'm not sure whether backups or restores for older Grafana versions would stop working with this update. Irrespective of that, we should implement functionality to skip specific types of resources for specific version. Then, we are 100% sure that this PR is no breaking change.

I thought of something like the following procedure.

1. The health endpoint returns the Grafana version.
2. We have a map that maps resource types to Grafana versions (e.g., library-elements: 8.4.0)
3. Based on 1. and 2., we decide whether to backup and restore a certain resource type.

What do you think?

Yeah sure, good idea. Could you help implement this ? Many thanks

[rubencosta]

@NilsLeger @ysde This approach would fail if there's no health endpoint which is the case for AWS managed grafana. See #184. If I might suggest, could the version be set as an env var possibly making things simpler on the implementation side and not relying on a health endpoint for getting the version? If we really want to get the version from the health endpoint then we would need a fallback if the health endpoint is not available as is the case with AWS managed grafana.

[nileger]

@rubencosta Great idea! Thanks for the input!

[azalenski-branch]

Is there something else that needs to be done in this PR? @ysde @rubencosta

If not is it possible to merge this PR soon?

[tedpearson]

Tried this out (merged into master) on grafana v9.3.1 for some library elements. Backing up seems to succeed, but restoring is failing with this error:

restoring library_element: /tmp/tmpvn9o2lwu/tmp/grafana-backup/library-elements/202212021805/2ShwS6zgz.library_element
[DEBUG] resp status: 200
[DEBUG] resp body: [{'id': 1, 'uid': 'MqUvRWWgz', 'title': 'Unifi'}]
query folder:, status:200
Traceback (most recent call last):
  File "/home/ted/.local/bin/grafana-backup", line 8, in <module>
    sys.exit(main())
  File "/home/ted/.local/lib/python3.9/site-packages/grafana_backup/cli.py", line 55, in main
    restore(args, settings)
  File "/home/ted/.local/lib/python3.9/site-packages/grafana_backup/restore.py", line 96, in main
    restore_components(args, settings, restore_functions, tmpdir)
  File "/home/ted/.local/lib/python3.9/site-packages/grafana_backup/restore.py", line 134, in restore_components
    restore_functions[ext](args, settings, file_path)
  File "/home/ted/.local/lib/python3.9/site-packages/grafana_backup/create_library_element.py", line 19, in main
    folder_id = get_folder(folder_uid, grafana_url, http_post_headers, verify_ssl, client_cert, debug)[1]['id']
TypeError: list indices must be integers or slices, not str

Here's the element it was trying to upload: element.txt

[tedpearson]

Looks like File "create_library_element.py", line 19 needs to read

folder_id = get_folder(folder_uid, grafana_url, http_post_headers, verify_ssl, client_cert, debug)[1][0]['id']

at least on my system.

Making just this change made it work 100% for me.

[maidanskyi]

This issue is still not fixed.

I have two AWS Managed Grafana: A and B. I did a backup for both of them and I'm trying to restore a data from A to Grafana B and I watch this error:

Traceback (most recent call last):
  File "/usr/bin/grafana-backup", line 33, in <module>
    sys.exit(load_entry_point('grafana-backup==1.2.5', 'console_scripts', 'grafana-backup')())
  File "/usr/lib/python3.10/site-packages/grafana_backup/cli.py", line 55, in main
    restore(args, settings)
  File "/usr/lib/python3.10/site-packages/grafana_backup/restore.py", line 96, in main
    restore_components(args, settings, restore_functions, tmpdir)
  File "/usr/lib/python3.10/site-packages/grafana_backup/restore.py", line 134, in restore_components
    restore_functions[ext](args, settings, file_path)
  File "/usr/lib/python3.10/site-packages/grafana_backup/create_library_element.py", line 19, in main
    folder_id = get_folder(folder_uid, grafana_url, http_post_headers, verify_ssl, client_cert, debug)[1]['id']

when I restore the original backup (B to B ) everything is ok