upgrade fail with error: unlinkat /var/lib/influxdb2: device or resource busy

[dlylei]

Hi all, When we upgrading influxDB from 1.7 to 2.0 by using InfluxDB v2 Helm chart with old data mounted, statefulsets deployed successful but the container in status of unready with Pod log below, could someone can help on this ? thank you in advance.

Error: couldn't delete existing file at "/var/lib/influxdb2": unlinkat /var/lib/influxdb2: device or resource busy See 'influxd -h' for help 2021-09-07T09:24:25. warn cleaning bolt and engine files to prevent conflicts on retry {"system": "docker", "bolt_path": "/var/lib/influxdb2/influxd.bolt", "engine_path": "/var/lib/influxdb2"} rm: can't remove '/var/lib/influxdb2': Resource busy

[doganyazar]

@dlylei I have the same problem. Did you figure out what the reason/solution is?

[William-LP]

@dlylei @doganyazar Hey guys, I'm facing that issue today. Did you manage to get over it?

[doganyazar]

@William-LP I actually did not experience this again so could not debug it. I guess it was a configuration problem on my side

[ccbur]

The problem is the parameter --overwrite-existing-v2 in the entrypoint.sh script. This parameter does not only "overwrite" an existing v2 installation. It tries to delete the root folder, too! But with the official helm chart the directory /var/lib/influxdb2 is a mount point and it can't be deleted.

Solution:

I used env variables in my values.yaml to move the directory one folder "deeper" for the upgrade:

env:
  - name: INFLUXD_BOLT_PATH
    value: /var/lib/influxdb2/influxdb2/influxd.bolt
  - name: INFLUXD_ENGINE_PATH
    value: /var/lib/influxdb2/influxdb2

With this /var/lib/influxdb2 is still a mount point, but /var/lib/influxdb2/influxdb2 is the root folder, which can be deleted. Voilà, works like a charm. After the upgrade I moved everything back to /var/lib/influxdb2 and removed those lines in my values.yaml.

Btw, I needed to set (temporarily) some initialDelaySeconds for the probes as the livenessprobe always killed my upgrade process. :-(