Open API for use with third party applications (Home Assistant, Node-RED, etc)

[skylord123]

Is your feature request related to a problem? Please describe. I want to use Node-RED to communicate with backrest via some sort of API.

Currently I have webhooks that fire off to Node-RED and I use that to send messages to my matrix room with updates about the backups. I want to be able to track the progress of the backup by making calls to some sort of endpoint to get information (similar to how the WebUI works currently).

This would also be really handy because we can build an integration for Home Assistant which would allow people to monitor/trigger their backups from their existing home automation platform.

Describe the solution you'd like Create an open API and document it so people can build cool stuff off it.

[balgerion]

Look at this: https://github.com/colby-int/restic-json

[skylord123]

Look at this: https://github.com/colby-int/restic-json

I know this exists. I want something that can query backrest since it already has all of this data available. Ideally the same info I can see via the web UI on backrest should also be available via API.

I did get the webhooks setup and working nicely but that is still just data being pushed out of backrest whereas I need to be able to query backrest for data directly.

For example: I would like to be able to query for info about a currently running backup to get the complete percentage and other info.

Another example: I want to setup a script to query the API and check previous backups to make sure all backups ran and completed successfully. Sort of like another tool that audits my backups to make sure things are running as it should. I currently have it message me every time a backup finishes but this is error prone since I have to manually verify them. The idea is to make this process automatic and an API would be perfect for this.

[garethgeorge]

It's great to see the interest in scripting / additional automation around backrest, this is certainly something I'd like to support. Putting out more formal docs covering Backrest's architecture and API are on my road map -- hopefully some of which can be code-generated. Today however you'll need to read the proto definitions to understand the API.

Today the API is defined formally as a gRPC service https://github.com/garethgeorge/backrest/blob/main/proto/v1/service.proto#L14-L63 which backrest makes accessible over an HTTP gateway which accepts binary requests OR protojson formatted requests. E.g.

curl -X POST 'localhost:9898/v1.Backrest/Backup' --data '{"value": "YOUR_PLAN_ID"}' -H 'Content-Type: application/json'

is a valid request to the Backup endpoint on the service to start a Backup.

Beyond this, for query support I suspect something like

curl -X POST 'localhost:9898/v1.Backrest/GetOperations' --data '{}' -H 'Content-Type: application/json'

should return the full operation history for your install, which may be quite a bit of JSON data. You can provide optional parameters i.e.

curl -X POST 'localhost:9898/v1.Backrest/GetOperations' --data '{"lastN": 1000, "selector": {"repoId": "myrepo"}}' -H 'Content-Type: application/json'

should return the last few thousand operations from the targeted repo myrepo. At the moment I expect that these APIs should be reasonably stable though it's possible that the the structures of the actual operations returned may change over time.

[skylord123]

@garethgeorge thank you! This is exactly what I needed.

Here is a flow for an example of how to do this in Node-RED (import using the hamburger menu in NR):

[{"id":"f8d0ed6480b4a91b","type":"inject","z":"2abaa0e7be1a6d3b","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"lastN\": 1000, \"selector\": {\"repoId\": \"repo-id\"}}","payloadType":"jsonata","x":250,"y":460,"wires":[["4f3d186ba804c950"]]},{"id":"4f3d186ba804c950","type":"http request","z":"2abaa0e7be1a6d3b","name":"","method":"POST","ret":"obj","paytoqs":"ignore","url":"http://localhost:9898/v1.Backrest/GetOperations","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"other","keyValue":"Content-Type","valueType":"other","valueValue":"application/json"}],"x":430,"y":460,"wires":[["900415374eb3824d"]]},{"id":"900415374eb3824d","type":"debug","z":"2abaa0e7be1a6d3b","name":"debug backrest api response","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":660,"y":460,"wires":[]}]

Now I am looking into creating a module for Node-RED to make integrating with backrest much easier for people in the future. Should make scripting a bit easier.

Keeping an eye out for the docs update :)

[Waler]

@skylord123 Is there any way to contribute to the Node-RED module? :)

[skylord123]

@Waler Sure! Here is a link: https://github.com/skylord123/node-red-backrest

It's still a WIP (only supports 3 endpoints currently) but feel free to contribute. I've just been working on it in my free time (haven't pushed to npm yet, just linking it locally for now).

[skylord123]

@garethgeorge is it possible to listen for events on /v1.Backrest/GetOperationEvents without protobuf? I've tried a few things to no avail.

[garethgeorge]

The http gateway is provided by connectrpc so https://connectrpc.com/docs/go/streaming/#http-representation may provide some help.

I think you may need to curl with the content-type set to application/grpc+json for streaming APIs.

[StevenMassaro]

I'll include the request I'm making to get the status of the most recent job for a particular plan, in case it helps anyone else:

curl -X POST 'localhost:9898/v1.Backrest/GetOperations' --data '{"lastN": 1, "selector": {"planId": "<my-plan-id>"}}' -H 'Content-Type: application/json'

[skylord123]

Just launched the first release of a Backrest module for Node-RED. I added a bunch of examples. Here is one of them that will start a backup and stream related operation events until it completes:

https://flows.nodered.org/node/@skylord123/node-red-contrib-backrest https://github.com/skylord123/node-red-backrest

Thanks @garethgeorge for answering all my questions.

[robflate]

Is there any progress on documenting or adapting the API? I'm looking for a general overview of the status of the latest backup completed on each plan. Similar to the data returned with /metrics. Currently, if I use something like;

curl -X POST 'localhost:9898/v1.Backrest/GetOperations' --data '{"selector": {"planId": "<my-plan-id>"}}' -H 'Content-Type: application/json'

The results are too busy to pull out the useful info. Similarly, if I limit the results with lastN": x I get the results of pending events also.

It would be great if we had an endpoint that just succinctly output the status of the most recent backup on each Plan.

[Fusion]

Is there any progress on documenting or adapting the API? I'm looking for a general overview of the status of the latest backup completed on each plan. Similar to the data returned with /metrics. Currently, if I use something like;

curl -X POST 'localhost:9898/v1.Backrest/GetOperations' --data '{"selector": {"planId": "<my-plan-id>"}}' -H 'Content-Type: application/json'

The results are too busy to pull out the useful info. Similarly, if I limit the results with lastN": x I get the results of pending events also.

It would be great if we had an endpoint that just succinctly output the status of the most recent backup on each Plan.

I have my endpoints listed in a toml file (parsed using a hideous bash parser) and iterate through them, extracting the status from each plan's output:

status=$(curl -u $user:$password -X POST "localhost:$port/v1.Backrest/GetOperations" --data "{\"lastN\": 1, \"selector\": {\"planId\": \"$planid\"}}" -H 'Content-Type: application/json' 2>/dev/null | jq -r '.operations[].status')