Automatic migration from InfluxDB OSS 1.8.10 to 2.1.1 fails

[PoloKltz]

Hi there,

I find it particularly difficult to follow the documentation to upgrade from 1.x to 2.x, and even after applying the required steps, I end up with fatal error or unauthorised access.

Below is a more precise description (at least I hope so) of the error and the steps to reproduce it

Minimal actions needed to reproduce the behavior.

1. Have a InfluxDB OSS 1.8.10 up and running on a single instance
2. Apply steps described to perform the upgrade
3. In particular use the following command to do the uprage

sudo -u influxdb influxd upgrade --config-file /etc/influxdb/influxdb.conf --v2-config-path /specific_path/influxdbv2/config.toml -e /specific_path/influxdbv2/engine -m /specific_path/influxdbv2/influxd.bolt -c /specific_path/influxdbv2/configs --log-path /specific_path/influxdbv2/upgrade.log --log-level debug 

4. Start the influxDB instance (as a service or as influxd process)

Expected behavior:

• Have documented and precise steps to follow to perform the upgrade:
• End up with an up and running InfluxDB 2.1.1 service, with all data from 1.8.10 migrated in bucket and organisation

Actual behavior: Differents errors when starting influxd :

• too many files open
• no org for default user
• Error: failed to list users: 401 Unauthorized: unauthorized access

Environment info:

• System info: Run Linux 5.13.0-1014-aws x86_64 20.04.1-Ubuntu
• InfluxDB version: InfluxDB v2.1.1

[lesam]

too many files open sounds like a bad ulimit setting, which could definitely make the upgrade fail (and might happen more commonly as users upgrade from inmem indexing on 1.x to TSI indexing in 2.x). I think we should add a warning for that.

[lesam]

@PoloKltz what is your ulimit -a setting? If you increase your file descriptor ulimit (depends on system, but something like ulimit -n unlimited) does that fix the errors?

[lesam]

Possibly related: https://github.com/influxdata/influxdb/issues/23157

[williamluke4]

Just run into the Error: 401 Unauthorized: unauthorized access after the 1.8 -> 2.3 upgrade

> ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 7801
max locked memory           (kbytes, -l) 253808
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 7801
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

> influx version
Influx CLI 2.3.0 (git: 88ba346) build_date: 2022-04-06T19:30:53Z

[williamluke4]

Managed to find the upgrade log

influxd upgrade
{"level":"info","ts":1656536691.8887737,"caller":"upgrade/upgrade.go:401","msg":"Starting InfluxDB 1.x upgrade"}
{"level":"info","ts":1656536691.889225,"caller":"upgrade/upgrade.go:404","msg":"Upgrading config file","file":"/etc/influxdb/influxdb.conf"}
{"level":"info","ts":1656536691.8897223,"caller":"upgrade/upgrade.go:408","msg":"Config file upgraded.","1.x config":"/etc/influxdb/influxdb.conf","2.x config":"/root/.influxdbv2/config.toml"}
{"level":"info","ts":1656536691.8898306,"caller":"upgrade/upgrade.go:418","msg":"Upgrade source paths","meta":"/var/lib/influxdb/meta","data":"/var/lib/influxdb/data"}
{"level":"info","ts":1656536691.889981,"caller":"upgrade/upgrade.go:419","msg":"Upgrade target paths","bolt":"/root/.influxdbv2/influxd.bolt","engine":"/root/.influxdbv2/engine"}
{"level":"info","ts":1656536691.8978586,"caller":"bolt/bbolt.go:83","msg":"Resources opened","service":"bolt","path":"/root/.influxdbv2/influxd.bolt"}
{"level":"info","ts":1656536691.8995106,"caller":"migration/migration.go:175","msg":"Bringing up metadata migrations","service":"migrations","migration_count":20}
> Welcome to InfluxDB 2.0!
? Please type your primary username grafana
? Please type your password *************
? Please type your password again *************
? Please type your primary organization name SwitchedOn
? Please type your primary bucket name primary
? Please type your retention period in hours, or 0 for infinite 0
? Setup with these parameters?
  Username:          grafana
  Organization:      SwitchedOn
  Bucket:            primary
  Retention Period:  infinite
 Yes
{"level":"info","ts":1656536738.6713157,"caller":"upgrade/setup.go:73","msg":"CLI config has been stored.","path":"/root/.influxdbv2/configs"}
{"level":"info","ts":1656536738.6717663,"caller":"upgrade/database.go:202","msg":"Checking available disk space"}
{"level":"info","ts":1656536738.683662,"caller":"upgrade/database.go:223","msg":"Computed disk space","free":"44 GB","required":"553 MB"}
? Proceeding will copy all V1 data to "/root/.influxdbv2"
  Space available: 44 GB
  Space required:  553 MB
 Yes
{"level":"info","ts":1656536745.3202834,"caller":"upgrade/database.go:51","msg":"Upgrading databases"}
{"level":"info","ts":1656536745.336234,"caller":"upgrade/database.go:101","msg":"Creating mapping","database":"home","retention policy":"autogen","orgID":"8404c2be38ca173b","bucketID":"73f2a66a585554cb"}
{"level":"info","ts":1656536748.8159993,"caller":"upgrade/database.go:195","msg":"Database upgrade complete","upgraded_count":1}
{"level":"info","ts":1656536748.8174427,"caller":"upgrade/security.go:45","msg":"Upgrading 1.x users"}
{"level":"warn","ts":1656536748.8176544,"caller":"upgrade/security.go:50","msg":"User is admin and will not be upgraded","username":"grafana"}
{"level":"info","ts":1656536748.8177702,"caller":"upgrade/security.go:115","msg":"User upgrade complete","upgraded_count":0}
{"level":"info","ts":1656536748.8178678,"caller":"upgrade/upgrade.go:488","msg":"Upgrade successfully completed. Start the influxd service now, then log in","login_url":"http://localhost:8086"}

[lesam]

@williamluke4 I think your issue is not the issue in the original ticket.

Just run into the Error: 401 Unauthorized: unauthorized access after the 1.8 -> 2.3 upgrade

Specifically, see https://docs.influxdata.com/influxdb/v2.3/upgrade/v1-to-v2/automatic-upgrade/#secure-by-default .

Before upgrading to 2.3, enable authentication in your InfluxDB 1.x instance and test your credentials to ensure your applications, agents, and visualization tools can connect to InfluxDB.

If you upgrade with auth-enabled = false, the upgrade may appear complete, but client requests to InfluxDB 2.3 may be silently ignored (you won’t see a notification the request was denied).

[samhld]

@williamluke4 Can you describe where and how you're getting the unauthorized error? I tried to reproduce this and the config that the setup provides me appropriate access. When I use a bad token, I get the expected errors.

The "config" I'm referring to you is the one created by this log line in the upgrade logs you pasted:

{"level":"info","ts":1656536738.6713157,"caller":"upgrade/setup.go:73","msg":"CLI config has been stored.","path":"/root/.influxdbv2/configs"}

This will set your CLI to use that configuration by default, which means it will also use an "operator" token created in that step by default. The operator token can do anything.