influxdb
influxdb copied to clipboard
Published
20 hours ago •
influxdata
influxdata
write points fail and report [shard 90] short buffer?
Steps to reproduce: List the minimal actions needed to reproduce the behavior.
- write point use java client
- client report
com.influxdb.exceptions.InternalServerErrorException: unexpected error writing points to database: [shard 90] short buffer
at com.influxdb.internal.AbstractRestClient.responseToError(AbstractRestClient.java:118) ~[influxdb-client-core-3.4.0.jar!/:3.4.0]
at com.influxdb.internal.AbstractRestClient.execute(AbstractRestClient.java:81) ~[influxdb-client-core-3.4.0.jar!/:3.4.0]
at com.influxdb.client.internal.AbstractWriteBlockingClient.write(AbstractWriteBlockingClient.java:85) ~[influxdb-client-java-3.4.0.jar!/:3.4.0]
at com.influxdb.client.internal.WriteApiBlockingImpl.lambda$writePoints$1(WriteApiBlockingImpl.java:137) ~[influxdb-client-java-3.4.0.jar!/:3.4.0]
- influxdb logs error too ,and only when write some data trigger this error
Environment info:
docker influx 2.1.1
Logs: Include snippet of errors in log.
2021-12-01T06:25:36.585734Z info Reading file {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "service": "cacheloader", "path": "/var/lib/influxdb2/engine/wal/cb8da52d13f762fb/autogen/96/_00001.wal", "size": 5712}
2021-12-01T06:25:36.591886Z info Opened file {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "service": "filestore", "path": "/var/lib/influxdb2/engine/data/cb8da52d13f762fb/autogen/94/000000001-000000001.tsm", "id": 0, "duration": "7.343ms"}
2021-12-01T06:25:36.611728Z info Opened file {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "service": "filestore", "path": "/var/lib/influxdb2/engine/data/cb8da52d13f762fb/autogen/89/000000003-000000002.tsm", "id": 0, "duration": "0.596ms"}
2021-12-01T06:25:43.419126Z error Failed to open shard {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "store", "op_name": "tsdb_open", "db_shard_id": 90, "error": "[shard 90] short buffer"}
2021-12-01T06:25:44.798705Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-01T06:25:44.808490Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-01T06:25:44.810225Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-01T06:25:44.851809Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-01T06:25:44.862204Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-01T07:00:56.567977Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:05:56.869878Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:03.029356Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:10.678191Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:10.694187Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:19.697167Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:19.706261Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:28.065021Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:28.074078Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:36.696444Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:36.705558Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:42.150872Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:42.159537Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:48.998667Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:49.007475Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:56.541653Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:08:56.550789Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:04.432700Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:04.441095Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:13.482655Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:13.492085Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:21.475071Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:21.483124Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-01T07:09:29.106772Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-02T02:55:50.189347Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-02T02:56:46.791163Z info Retention policy deletion check (start) {"log_id": "0Y9AZ_e0000", "service": "retention", "op_name": "retention_delete_check", "op_event": "start"}
2021-12-02T02:56:46.791346Z info Retention policy deletion check (end) {"log_id": "0Y9AZ_e0000", "service": "retention", "op_name": "retention_delete_check", "op_event": "end", "op_elapsed": "0.206ms"}
2021-12-02T02:58:51.197395Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-02T03:00:50.281280Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-02T03:03:44.792549Z info Cache snapshot (start) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "start"}
2021-12-02T03:03:44.824314Z info Snapshot for path written {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "path": "/var/lib/influxdb2/engine/data/d985752f6c791b5b/autogen/20", "duration": "31.791ms"}
2021-12-02T03:03:44.824357Z info Cache snapshot (end) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "end", "op_elapsed": "31.833ms"}
2021-12-02T03:03:49.558056Z info Cache snapshot (start) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "start"}
2021-12-02T03:03:49.588914Z info Snapshot for path written {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "path": "/var/lib/influxdb2/engine/data/592474454cfee085/autogen/99", "duration": "30.889ms"}
2021-12-02T03:03:49.588946Z info Cache snapshot (end) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "end", "op_elapsed": "30.919ms"}
2021-12-02T03:03:53.616409Z info Cache snapshot (start) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "start"}
2021-12-02T03:03:53.648065Z info Snapshot for path written {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "path": "/var/lib/influxdb2/engine/data/e801bb6a29638f3e/autogen/100", "duration": "31.674ms"}
2021-12-02T03:03:53.648100Z info Cache snapshot (end) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "end", "op_elapsed": "31.709ms"}
2021-12-02T03:04:25.317616Z info Unauthorized {"log_id": "0Y9AZ_e0000", "error": "token required"}
2021-12-02T03:04:55.852016Z info Execute source panic {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 776506 [running]:
runtime/debug.Stack()
/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
/home/circleci/go/pkg/mod/github.com/influxdata/[email protected]/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc00f43eed0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc0587a6540}, 0xc0617efbd0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc0587a6540}, 0xc00f43e7b0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc01c044180, 0x61a837d7)
/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc00fa53860, {0x40a6b18, 0xc0587a6540}, {0x406f510, 0xc01c044180}, 0xc016b2c0b0)
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc00fa53860, {0x40a6b18, 0xc0587a6540})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc00fa53860, {0x40a6b18, 0xc0587a6540})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc00fa53860})
/home/circleci/go/pkg/mod/github.com/influxdata/[email protected]/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
/home/circleci/go/pkg/mod/github.com/influxdata/[email protected]/execute/executor.go:261 +0x228
2021-12-02T03:04:57.479246Z info Execute source panic {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 776589 [running]:
runtime/debug.Stack()
/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
/home/circleci/go/pkg/mod/github.com/influxdata/[email protected]/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc01d85f710})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc015cda100}, 0xc01758fbd0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc015cda100}, 0xc01d85f020)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc01d839380, 0x61a837d9)
/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc00b639a00, {0x40a6b18, 0xc015cda100}, {0x406f510, 0xc01d839380}, 0xc03c8df0e0)
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc00b639a00, {0x40a6b18, 0xc015cda100})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc00b639a00, {0x40a6b18, 0xc015cda100})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc00b639a00})
/home/circleci/go/pkg/mod/github.com/influxdata/[email protected]/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
/home/circleci/go/pkg/mod/github.com/influxdata/[email protected]/execute/executor.go:261 +0x228
2021-12-02T03:04:59.934067Z info Execute source panic {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 776621 [running]:
runtime/debug.Stack()
/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
/home/circleci/go/pkg/mod/github.com/influxdata/[email protected]/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc01e71b800})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc0138f7340}, 0xc03d883bd0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc0138f7340}, 0xc01e71b110)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc01e70d100, 0x61a837db)
/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc0181f2d00, {0x40a6b18, 0xc0138f7340}, {0x406f510, 0xc01e70d100}, 0xc07acbf840)
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc0181f2d00, {0x40a6b18, 0xc0138f7340})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc0181f2d00, {0x40a6b18, 0xc0138f7340})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc0181f2d00})
/home/circleci/go/pkg/mod/github.com/influxdata/[email protected]/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
/home/circleci/go/pkg/mod/github.com/influxdata/[email protected]/execute/executor.go:261 +0x228
2021-12-02T03:05:50.338055Z info Write failed {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "service": "write", "shard": 90, "error": "[shard 90] short buffer"}
2021-12-02T03:08:09.183024Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-02T03:08:09.183373Z info Reindexing TSM data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 101}
2021-12-02T03:08:09.183392Z info Reindexing WAL data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 101}
2021-12-02T03:09:00.159441Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-02T03:09:00.159977Z info Reindexing TSM data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 102}
2021-12-02T03:09:00.159995Z info Reindexing WAL data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 102}
2021-12-02T03:09:54.759458Z info index opened with 8 partitions {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "index": "tsi"}
2021-12-02T03:09:54.759771Z info Reindexing TSM data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 103}
2021-12-02T03:09:54.759784Z info Reindexing WAL data {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "db_shard_id": 103}
2021-12-02T03:14:03.878068Z info Execute source panic {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 903477 [running]:
runtime/debug.Stack()
/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
/home/circleci/go/pkg/mod/github.com/influxdata/[email protected]/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc00b13f740})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc01c242cc0}, 0xc028b75bd0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc01c242cc0}, 0xc00b13f5f0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc00257e380, 0x61a839fb)
/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc031c8c410, {0x40a6b18, 0xc01c242cc0}, {0x406f510, 0xc00257e380}, 0xc0d402d520)
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc031c8c410, {0x40a6b18, 0xc01c242cc0})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc031c8c410, {0x40a6b18, 0xc01c242cc0})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc031c8c410})
/home/circleci/go/pkg/mod/github.com/influxdata/[email protected]/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
/home/circleci/go/pkg/mod/github.com/influxdata/[email protected]/execute/executor.go:261 +0x228
2021-12-02T03:14:07.591332Z info Execute source panic {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 903622 [running]:
runtime/debug.Stack()
/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
/home/circleci/go/pkg/mod/github.com/influxdata/[email protected]/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc01c3ea150})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc01cf75240}, 0xc016bf8bd0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc01cf75240}, 0xc01c3ea000)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc01b72c400, 0x61a839ff)
/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc031c8cc30, {0x40a6b18, 0xc01cf75240}, {0x406f510, 0xc01b72c400}, 0xc0116b95b0)
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc031c8cc30, {0x40a6b18, 0xc01cf75240})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc031c8cc30, {0x40a6b18, 0xc01cf75240})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc031c8cc30})
/home/circleci/go/pkg/mod/github.com/influxdata/[email protected]/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
/home/circleci/go/pkg/mod/github.com/influxdata/[email protected]/execute/executor.go:261 +0x228
2021-12-02T03:14:14.450359Z info Execute source panic {"log_id": "0Y9AZ_e0000", "service": "storage-reads", "error": "panic: runtime error: invalid memory address or nil pointer dereference"}
goroutine 904050 [running]:
runtime/debug.Stack()
/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65
github.com/influxdata/flux/execute.(*executionState).do.func1.1()
/home/circleci/go/pkg/mod/github.com/influxdata/[email protected]/execute/executor.go:286 +0x205
panic({0x3c52600, 0x53e34d0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields.func1()
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:546 +0x1c
panic({0x3c7bc60, 0xc01d8e8db0})
/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).measurementFields(0xc011caaba0, {0x40a6b18, 0xc01daf9340}, 0xc002a4abd0)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:549 +0x33f
github.com/influxdata/influxdb/v2/v1/services/storage.(*Store).TagValues(0x4079ad0, {0x40a6b18, 0xc01daf9340}, 0xc01d8e8c60)
/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/storage/store.go:406 +0x390
github.com/influxdata/influxdb/v2/storage/flux.(*tagValuesIterator).Do(0xc01d496800, 0x61a83a06)
/home/circleci/go/src/github.com/influxdata/influxdb/storage/flux/reader.go:970 +0x1c6
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc0173ff040, {0x40a6b18, 0xc01daf9340}, {0x406f510, 0xc01d496800}, 0xc02bd34880)
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:76 +0xbf
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readTagValuesSource).run(0xc0173ff040, {0x40a6b18, 0xc01daf9340})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:492 +0x114
github.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc0173ff040, {0x40a6b18, 0xc01daf9340})
/home/circleci/go/src/github.com/influxdata/influxdb/query/stdlib/influxdata/influxdb/source.go:56 +0x168
github.com/influxdata/flux/execute.(*executionState).do.func1({0x40a86e0, 0xc0173ff040})
/home/circleci/go/pkg/mod/github.com/influxdata/[email protected]/execute/executor.go:291 +0x20e
created by github.com/influxdata/flux/execute.(*executionState).do
/home/circleci/go/pkg/mod/github.com/influxdata/[email protected]/execute/executor.go:261 +0x228
2021-12-02T03:15:51.054091Z info Cache snapshot (start) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "start"}
2021-12-02T03:15:51.057282Z info Snapshot for path written {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "path": "/var/lib/influxdb2/engine/data/cb8da52d13f762fb/autogen/98", "duration": "3.215ms"}
2021-12-02T03:15:51.057308Z info Cache snapshot (end) {"log_id": "0Y9AZ_e0000", "service": "storage-engine", "engine": "tsm1", "op_name": "tsm1_cache_snapshot", "op_event": "end", "op_elapsed": "3.238ms"}
2021-12-02T03:17:16.205704Z warn internal error not returned to client {"log_id": "0Y9AZ_e0000", "handler": "error_logger", "error": "context canceled"}
2021-12-02T03:26:46.791140Z info Retention policy deletion check (start) {"log_id": "0Y9AZ_e0000", "service": "retention", "op_name": "retention_delete_check", "op_event": "start"}
2021-12-02T03:26:46.791292Z info Retention policy deletion check (end) {"log_id": "0Y9AZ_e0000", "service": "retention", "op_name": "retention_delete_check", "op_event": "end", "op_elapsed": "0.164ms"}
FYI, we are seeing this as well. InfluxDB v2.1.1, Python client. Writing a large number of rows, we first see the issue on row 14995. The error definitely appears to be data-dependent. From skimming the InfluxDB source, this comes from extracting an array of strings from a compressed sequence of bytes. Apparently something becomes corrupt and the decoder expects to find more bytes of string than are actually present.
Going to see if I can downgrade to an earlier InfluxDB and check against that.
In our case, this appears to be caused by corruption in the bucket we're writing to. I've confirmed that writing to a new bucket in a different org works, and even a new bucket in the same org. But writing to this particular bucket/org fails every time, regardless of whether that particular measurement has been purged or not.
@abelletti You're right, I can fix this issue by create a new bucket, thanks. Hope IInfluxdata can add some feature to fix corrupted data in bucket.
We have seen this when a disk fills up. That can cause corrupt files which persist after a disk resize or move to a larger disk.
