Influx started from helm does not show data of migrated database

[MoJo2600]

I recently upgraded my influx 1.8 to influx 2.2. I used the way described in the documentation:

• Mount the v1 data to /var/lib/influxdb
• Mount a empty v2 data directory to /var/lib/influxdb2
• Start the container with the env variable DOCKER_INFLUXDB_INIT_MODE=upgrade

The data is upgraded and when the migration is finished I can access the new data just fine. I then sopped the migration container and configured the helm chart to use the newly created v2 data pvc. The container did start and I can see the buckets, but I don't see any measurements. The error I received was:

error="panic: runtime error: invalid memory address or nil pointer dereference" stacktrace="goroutine 1605 [running]:\nruntime/debug.Stack()\n\t/go/src/runtime/debug/stack.go:24 +0x65\ngithub.com/influxdata/flux/execute.(*executionState).recover(0xc001886000)\n\t/go/pkg/mod/github.com/in

I checked everything and after I compared the configuration of the migration container to the container startet with helm, I saw that the setting engine-path differs. In the helm chart container it is set to /var/lib/influxdb2 and in the migration container it is set to /var/lib/influxdb2/engine. After adding a new environment variable to my values.yaml likes this everything was working:

env:
  - name: INFLUXD_ENGINE_PATH
    value: /var/lib/influxdb2/engine

What makes me wonder: If the migration container from to official docker image creates the engine folder under /var/lib/influxdb2/engine it might be, that the configuration in this helm chart is wrong?

So in statefulset.yaml it is right now:

            # Path to persistent storage engine files where InfluxDB stores all Time-Structure Merge Tree (TSM) data on disk.
            - name: INFLUXD_ENGINE_PATH
              value: {{ .Values.persistence.mountPath }}

but it should be:

            # Path to persistent storage engine files where InfluxDB stores all Time-Structure Merge Tree (TSM) data on disk.
            - name: INFLUXD_ENGINE_PATH
              value: {{ .Values.persistence.mountPath }}/engine

What do you think?

[alespour]

I think you are right.

A fix must not break existing deployments though, so maybe something like

              value: {{ .Values.persistence.mountPath }}/{{ .Values.persistence.engineSubPath }}

with values.yaml

persistence:
  # workaround for issue #474
  # use empty string for fresh 2.x installations, "engine" for upgrades from 1.x if you have problem accessing migrated data
  engineSubPath: ""

could work?

[MoJo2600]

I can create a pull request for this. But a non breaking change would mean, that the existing installations are "wrong". I'm not absolutely sure what the path would be on a non docker installation. If the influx default would be /engine one might have to think about a form of migration to fix this issue?

[alespour]

Yeah, they are "wrong", the file system layout does not match the documentation, and a good practice probably.

I have no idea what migration to fix this could look like, and if it is worth the effort, though. I kind of look at this now as a known issue candidate :slightly_smiling_face:

[alespour]

Just for reference: https://docs.influxdata.com/influxdb/v2.2/reference/internals/file-system-layout/

[MoJo2600]

I'll have a look... maybe its doable.