panic: runtime error: slice bounds out of range

[oyearunpal]

Due to 100% disk space utilization my influxdb got corrupted and I tried various method to make it live however we are always facing issue of "slice bounds out of range" Steps to reproduce: docker run -p 8086:8086 -v /db_data/influxdb:/var/lib/influxdb influxdb:1.8

Expected behavior: It should up normally and start serving the data.

Actual behavior: Throwing errors

.......
.......
ts=2022-11-25T07:03:22.907711Z lvl=info msg="Finished compacting files" log_id=0eNPMdsG000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0eNPP_dW000 op_name=tsm1_compact_group db_shard_id=2876 tsm1_files_n=1
ts=2022-11-25T07:03:22.907718Z lvl=info msg="TSM compaction (end)" log_id=0eNPMdsG000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0eNPP_dW000 op_name=tsm1_compact_group db_shard_id=2876 op_event=end op_elapsed=6521.536ms
ts=2022-11-25T07:03:30.062966Z lvl=info msg="Compacted file" log_id=0eNPMdsG000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0eNPPZ~l000 op_name=tsm1_compact_group db_shard_id=2888 tsm1_index=0 tsm1_file=/var/lib/influxdb/data/_internal/monitor/2888/000000053-000000002.tsm.tmp
ts=2022-11-25T07:03:30.063000Z lvl=info msg="Finished compacting files" log_id=0eNPMdsG000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0eNPPZ~l000 op_name=tsm1_compact_group db_shard_id=2888 tsm1_files_n=1
ts=2022-11-25T07:03:30.063007Z lvl=info msg="TSM compaction (end)" log_id=0eNPMdsG000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0eNPPZ~l000 op_name=tsm1_compact_group db_shard_id=2888 op_event=end op_elapsed=13839.343ms
ts=2022-11-25T07:03:30.199978Z lvl=warn msg="Mixed shard index types" log_id=0eNPMdsG000 service=store tsi1_count=15 inmem_count=2 db_instance=_internal
ts=2022-11-25T07:03:31.878323Z lvl=info msg="TSM compaction (end)" log_id=0eNPMdsG000 engine=tsm1 tsm1_level=2 tsm1_strategy=level trace_id=0eNPPZiW000 op_name=tsm1_compact_group db_shard_id=2542 op_event=end op_elapsed=15727.945ms
panic: runtime error: slice bounds out of range [:23849] with capacity 158

goroutine 13881 [running]:
github.com/influxdata/influxdb/tsdb/cursors.(*IntegerArray).Include(0xc03956cc38, 0x16f7268b6b86d618, 0x16f7280578ec28c8)
        /go/src/github.com/influxdata/influxdb/tsdb/cursors/arrayvalues.gen.go:327 +0x25c
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*tsmBatchKeyIterator).combineInteger(0xc0e243e000, 0xc07e1ee301, 0xb, 0xc07e1ee3a0, 0x1)
        /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.gen.go:1339 +0x2f3
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*tsmBatchKeyIterator).mergeInteger(0xc0e243e000)
        /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.gen.go:1282 +0x193
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*tsmBatchKeyIterator).merge(0xc0e243e000)
        /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:1850 +0x66
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*tsmBatchKeyIterator).Next(0xc0e243e000, 0x7f55093dea96)
        /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:1716 +0x13eb
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).write(0xc03d8d2120, 0xc03c03b540, 0x4b, 0x25aa640, 0xc0e243e000, 0xc0b5573b01, 0x0, 0x0)
        /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:1141 +0x1dd
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).writeNewFiles(0xc03d8d2120, 0x1fa, 0x3, 0xc0826a38c0, 0x4, 0x4, 0x25aa640, 0xc0e243e000, 0x1, 0x25aa640, ...)
        /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:1045 +0x18f
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).compact(0xc03d8d2120, 0xc0826a3800, 0xc0826a38c0, 0x4, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:953 +0x40d
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).CompactFull(0xc03d8d2120, 0xc0826a38c0, 0x4, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:971 +0x17e
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*compactionStrategy).compactGroup(0xc0bc6ca850)
        /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2289 +0x1c72
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*compactionStrategy).Apply(0xc0bc6ca850)
        /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2266 +0x4d
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactHiPriorityLevel.func1(0xc0152ce430, 0xc0301acc80, 0x2, 0xc0bc6ca850)
        /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2183 +0x12e
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactHiPriorityLevel
        /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2178 +0x123

Environment info:

• System info: Linux 3.10.0-1160.el7.x86_64 x86_64

Config: Default one

[davidby-influx]

This does appear to be file corruption. IF you have bad files, it is difficult for code to operate correctly.