Silent flux failure while using to(), "internal error not returned to client"

[nat45928]

I am attempting to side load ~2GB worth of backed up data from a temporary bucket to an existing bucket in my InfluxDB instance. To do this I am running the following flux query:

      from(bucket: "tmp")
      |> range(start: -1y)
      |> filter(fn: (r) => r._measurement !~ /event_logs/)
      |> to(bucket: "dest")
      |> map(fn: (r) => ({"_value": 1, "_time": r._time, "_measurement": r._measurement}))
      |> group(columns: ["_measurement"])
      |> count()

While running this query it sometimes appears the influx query CLI command hangs. During this I can see my instances memory usage and CPU usage drop (as if the query stopped, the influx instance usually maxes a single core and uses >8Gi of memory during this time) and I find this entry in the logs:

ts=2023-01-20T16:44:26.130913Z lvl=warn msg="internal error not returned to client" log_id=0fTme62G000
handler=error_logger error="context canceled"

Any ideas on how to make this restore process more reliable (and possibly faster)?

Steps to reproduce: List the minimal actions needed to reproduce the behavior.

1. influx backup ~2GiB worth of data from a single bucket
2. influx restore into a temp bucket
3. Attempt to use Flux to sideload points into an existing bucket with data in it.

Expected behavior: Sideloading to complete.

Actual behavior: Indefinite hang.

Environment info:

• System info: Linux 5.4.0-1098-azure x86_64 - Azure AKS, Standard_E4as_v4 node
• InfluxDB version: InfluxDB v2.6.0 (git: 24a2b621ea) build_date: 2022-12-15T18:47:00Z
• Other relevant environment details: N/A

Config:

{
        "assets-path": "",
        "bolt-path": "/var/lib/influxdb2/influxd.bolt",
        "e2e-testing": false,
        "engine-path": "/var/lib/influxdb2",
        "feature-flags": null,
        "flux-log-enabled": true,
        "hardening-enabled": false,
        "http-bind-address": ":8086",
        "http-idle-timeout": 86400000000000,
        "http-read-header-timeout": 10000000000,
        "http-read-timeout": 0,
        "http-write-timeout": 0,
        "influxql-max-select-buckets": 0,
        "influxql-max-select-point": 0,
        "influxql-max-select-series": 0,
        "instance-id": "",
        "log-level": "info",
        "metrics-disabled": false,
        "nats-max-payload-bytes": 0,
        "nats-port": 4222,
        "no-tasks": false,
        "pprof-disabled": false,
        "query-concurrency": 1024,
        "query-initial-memory-bytes": 0,
        "query-max-memory-bytes": 0,
        "query-memory-bytes": 0,
        "query-queue-size": 1024,
        "reporting-disabled": false,
        "secret-store": "bolt",
        "session-length": 1440,
        "session-renew-disabled": false,
        "sqlite-path": "/var/lib/influxdb2/influxd.sqlite",
        "storage-cache-max-memory-size": 1073741824,
        "storage-cache-snapshot-memory-size": 26214400,
        "storage-cache-snapshot-write-cold-duration": "10m0s",
        "storage-compact-full-write-cold-duration": "4h0m0s",
        "storage-compact-throughput-burst": 50331648,
        "storage-max-concurrent-compactions": 0,
        "storage-max-index-log-file-size": 1048576,
        "storage-no-validate-field-size": false,
        "storage-retention-check-interval": "30m0s",
        "storage-series-file-max-concurrent-snapshot-compactions": 0,
        "storage-series-id-set-cache-size": 0,
        "storage-shard-precreator-advance-period": "30m0s",
        "storage-shard-precreator-check-interval": "10m0s",
        "storage-tsm-use-madv-willneed": false,
        "storage-validate-keys": false,
        "storage-wal-fsync-delay": "0s",
        "storage-wal-max-concurrent-writes": 0,
        "storage-wal-max-write-delay": 600000000000,
        "storage-write-timeout": 10000000000,
        "store": "disk",
        "testing-always-allow-setup": false,
        "tls-cert": "",
        "tls-key": "",
        "tls-min-version": "1.2",
        "tls-strict-ciphers": false,
        "tracing-type": "",
        "ui-disabled": false,
        "vault-addr": "",
        "vault-cacert": "",
        "vault-capath": "",
        "vault-client-cert": "",
        "vault-client-key": "",
        "vault-client-timeout": 0,
        "vault-max-retries": 0,
        "vault-skip-verify": false,
        "vault-tls-server-name": "",
        "vault-token": ""
}

[jeffreyssmith2nd]

Do you ever see any data show up in the new bucket or does all the data from tmp stay missing?

A workaround you could try is to use the stop parameter to the range and write multiple smaller batches.

[nat45928]

Yep, using the stop parameter to create smaller chunks in sequence has been our workaround. I can't say we've investigated the resulting state of the bucket, I'll report back.

The bigger problem for us is that the HTTP response is never sent and the client appears to hang, so our scripts spin forever before timing out. Do you have any methods of debugging why that might be the case?

[utdrmac]

1+ year old issue. any insights? I'm getting this same error when doing 10KB data loads from Telegraf.