influxdb server crashed when reading a corrupt file

[CoderSong2015]

Steps to reproduce: List the minimal actions needed to reproduce the behavior.

It‘s not easy to procedure because it is caused by file system error

Expected behavior: Successfully reading the corrupt tsm file and return error info

Actual behavior: the server crashed and throw slice bounds out of range error

Environment info:

• System info: centos 7
• InfluxDB version: from 1.7 to 2.0

Logs: Include snippet of errors in log. s=2020-10-30T15:06:52.317091Z lvl=info msg="Opened file" log_id=0QAWWAwG000 engine=tsm1 service=filestore path=XXX/autogen/45/000000072-000000002.tsm id=12 duration=74931.498ms panic: runtime error: slice bounds out of range

goroutine 9075 [running]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*indirectIndex).UnmarshalBinary(0xc072082b40, 0xc5c0000000, 0x703f04fe, 0x703f04fe, 0x0, 0x0)

I have found the crash reason： In the function indirectIndex.UnmarshalBinary of reader.go ` func (d *indirectIndex) UnmarshalBinary(b []byte) error {...

for i < iMax {
	...
    //1、 count maybe get a value zero
    count := int32(binary.BigEndian.Uint16(b[i : i+indexCountSize]))
    ...
     //2、 and then i will be a negative number
    ...
     //3、slice bounds out of range here
    maxT := int64(binary.BigEndian.Uint64(b[i+8 : i+16]))
    if maxT > maxTime {
         maxTime = maxT
    }
    i += indexEntrySize

} ...`

[CoderSong2015]

？

[gggevorgyan]

Hi Is there any updates related to this issue? I have faced to similar situation:

Steps to reproduce: List the minimal actions needed to reproduce the behavior. I can provide problematic tsm file, if it will speed up investigation process.

Expected behavior: Rename corrupted tsm file as it is expected to be done other cases.

Actual behavior: the server crashed and throw slice bounds out of range error

Environment info: System info: Ubuntu 18.04 InfluxDB version: from 1.8.5, integrated in code. But using influx_inpect also crashes same way.

Logs:

`panic: runtime error: slice bounds out of range [:-7]

goroutine 1275 [running]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*indirectIndex).UnmarshalBinary(0xc00edd13b0, 0x7edba47377c0, 0x8ec4e, 0x8ec56, 0x0, 0x0) /var/vcap/data/compile/log-store/vendor/github.com/influxdata/influxdb/tsdb/engine/tsm1/reader.go:1254 +0x807 github.com/influxdata/influxdb/tsdb/engine/tsm1.(*mmapAccessor).init(0xc02cb3ef60, 0x0, 0x0, 0x0) /var/vcap/data/compile/log-store/vendor/github.com/influxdata/influxdb/tsdb/engine/tsm1/reader.go:1359 +0x29c github.com/influxdata/influxdb/tsdb/engine/tsm1.NewTSMReader(0xc00020aa90, 0xc078315a90, 0x1, 0x1, 0x0, 0x0, 0x0) /var/vcap/data/compile/log-store/vendor/github.com/influxdata/influxdb/tsdb/engine/tsm1/reader.go:239 +0x19c github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Open.func1(0xc00a2d4c30, 0xc0749c7980, 0x3, 0xc00020aa90) /var/vcap/data/compile/log-store/vendor/github.com/influxdata/influxdb/tsdb/engine/tsm1/file_store.go:543 +0x11d created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Open /var/vcap/data/compile/log-store/vendor/github.com/influxdata/influxdb/tsdb/engine/tsm1/file_store.go:535 +0x44a `

Tried to run influx_inspect on that but received similar crash there as well.

`panic: runtime error: slice bounds out of range [:-7]

goroutine 1 [running]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*indirectIndex).UnmarshalBinary(0xc00012ae10, 0x7f30a0cb17c0, 0x8ec4e, 0x8ec56, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/reader.go:1254 +0x78f github.com/influxdata/influxdb/tsdb/engine/tsm1.(*mmapAccessor).init(0xc0000c84e0, 0x0, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/reader.go:1359 +0x2a2 github.com/influxdata/influxdb/tsdb/engine/tsm1.NewTSMReader(0xc000010010, 0x0, 0x0, 0x0, 0xc000010010, 0x0, 0x0) /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/reader.go:239 +0x190 github.com/influxdata/influxdb/cmd/influx_inspect/verify/tsm.(*verifyTSM).TSMReader(0xc0000c8f60, 0x0, 0x0, 0xc) /go/src/github.com/influxdata/influxdb/cmd/influx_inspect/verify/tsm/verify.go:122 +0xbf github.com/influxdata/influxdb/cmd/influx_inspect/verify/tsm.(*verifyChecksums).Run(0xc0000c8f60, 0xca2c00, 0xc0000f4fd0, 0xc0001342a0, 0x2a, 0x0, 0x9) /go/src/github.com/influxdata/influxdb/cmd/influx_inspect/verify/tsm/verify.go:154 +0x519 github.com/influxdata/influxdb/cmd/influx_inspect/verify/tsm.(*Command).Run(0xc0000dac20, 0xc0000cc060, 0x2, 0x2, 0x0, 0xc0000cc060) /go/src/github.com/influxdata/influxdb/cmd/influx_inspect/verify/tsm/verify.go:58 +0x3b4 main.(*Main).Run(0xc000099f18, 0xc0000cc060, 0x2, 0x2, 0x0, 0x4) /go/src/github.com/influxdata/influxdb/cmd/influx_inspect/main.go:108 +0x83e main.main() /go/src/github.com/influxdata/influxdb/cmd/influx_inspect/main.go:28 +0x151`

[Raj0564]

@here, any update on this?

[gggevorgyan]

@here Could you please provide update on this issue? Do you need more info, or is there missing part in this issue, thats why you don't pay attention on this ?

[gggevorgyan]

@here Is there any updated, related to this issue?