Unable to restore backup to existing bucket

[howardtopher]

Steps to reproduce:

1. Create backup # influx backup -b bucket1 ~/influxdb-v2-export/
2. Copy files to different server where bucket already exists with data in it
3. Attempt to restore # influx restore -b bucket1 ~/influxdb-v2-export/

Expected behavior:

Data is restored to existing bucket along side the already existing data in that bucket.

Documentation (https://docs.influxdata.com/influxdb/v2.0/reference/cli/influx/restore/) provides an example for restoring data to an existing bucket separate from restoring to a new bucket.

Actual behavior:

Errors out saying "cannot create bucket".

2021-02-15T19:34:58.266935Z	info	Resources opened	{"log_id": "0SLnrsqW000", "path": "influxdb-v2-export/20210215T185834Z.bolt"}
2021-02-15T19:34:58.267266Z	info	Restoring organization	{"log_id": "0SLnrsqW000", "id": "583d63a670ab779c", "name": "org1"}
2021-02-15T19:34:58.268984Z	info	Restoring bucket	{"log_id": "0SLnrsqW000", "id": "04ea7e91e2f1a222", "name": "bucket1"}
Error: Cannot create bucket: bucket with name bucket1 already exists.
See 'influx restore -h' for help

Environment info:

• System info: Linux 4.18.0-277.el8.x86_64 x86_64
• InfluxDB version: InfluxDB 2.0.3 (git: fe04d346df) build_date: 2020-12-15T01:00:16Z
• Other relevant environment details: Source and destination systems are running identical version of InfluxDB

[afzaal-ameer]

Any known workarounds for this ?

[howardtopher]

Any known workarounds for this ?

Here's what I did to get around it. I was migrating from an old server to a new server and to prevent having a gap in data the new server started logging into a fresh bucket and my intent was to just restore the old data into that bucket. According to the documentation that should work, but obviously there's either a bug or incorrect docs.

1. Stop telegraf or any other software you're using to put data into influxdb (just for a minute)
2. Rename the bucket in influxdb to something temporary.
3. Change telegraf and any other software to use that temporary bucket name.
4. Restart telegraf and others to keep gathering data.

This frees up the bucket name you want to restore and allows you to still capture data so you don't have a gap. Then,

1. Restore the backup of the bucket.
2. Undo the changes from above. (change telegraf and others back to the original bucket name)

After this you have all the old data and any new data from this moment on in the bucket that you've restored. To get the data from the temporary bucket to the restored one, do a query/write.

1. influx query 'from(bucket: "tempbucket") |> range(start: -10d)' --raw | tee /dev/tty | influx write --bucket originalbucket --format csv --rate-limit 20MB/min

Make sure the range goes back far enough to cover your needs. I included a tee in there so I could watch progress as it went through all the measurements. Also a rate limit to prevent knocking the influxdb process down (it did in my testing). Once done, you can delete the temporary bucket. All data exists in the single bucket that you originally wanted to restore to and you don't have any gaps in data.

This took me a couple days of trying different things to get it figured out. You could make it easier and just restore your bucket to a different bucket name and then query/write that one over to the correct bucket, but in my case I had 6 years of data to restore and that would have taken a very long time. It was much quicker free up the bucket name, restore the old data, and then query/write only the few days of data I had stored since moving to the new server.

[danxmoran]

@howardtopher (and others hitting this): I took a stab at a fix for this, but walked it back for now. The server-side restore API allows restoring an existing bucket but doing so causes any existing data to be deleted before the backed-up data is loaded. Changing the behavior to merge old&new shards will require a more extensive change.

In the meantime, I'll make sure our docs are fixed. Thank you for posting your work-around, we'll incorporate a version of it in the updated docs.

[isarrider]

Hi @danxmoran - is there any update? I guess many people are looking for a merging feature...

[neilbalch]

Yes, me too. This is a surprisingly missing feature

[zachibs]

Hi, are there any plans to make this a feature? didn't seem to find any way to restore data into an existing bucket.

[ckuhtz]

I need this as well. :/ Any updates beside the workaround?

[hafiz-usman-cowlar]

Use --full flag after path, to completely restore new database

[neilbalch]

Use --full flag after path, to completely restore new database

This is not a solution. I've tried that and --full replaces EVERYTHING about the InfluxDB instance, screwing up the local configuration and (in my experience) corrupting the entire local copy with significant data loss.