minio
crash with high concurrency in warp put
warp put --tls --insecure --host 10.10.100.61:9000 --access-key minio --secret-key minio123 --autoterm --concurrent 168
panic: runtime error: slice bounds out of range [24560:16400]▓▓▓█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░┃ 6.67%
goroutine 953 [running]:
github.com/secure-io/sio-go.(*EncReader).Read(0xc00033ebb0, {0xc002394000?, 0xc0072929a8?, 0x41d4f6?})
github.com/secure-io/[email protected]/reader.go:57 +0x1ea
io.ReadAtLeast({0xe309e0, 0xc00033ebb0}, {0xc002394000, 0x2000, 0x2000}, 0x2000)
io/io.go:335 +0x90
io.ReadFull(...)
io/io.go:354
github.com/minio/warp/pkg/generator.(*scrambler).Read(0xc000640090, {0xc002394000?, 0x452ae9?, 0x2000?})
github.com/minio/warp/pkg/generator/scambler.go:116 +0x6c
github.com/minio/minio-go/v7.(*hookReader).Read(0xc002c2aa80, {0xc002394000, 0x6?, 0x2000})
github.com/minio/minio-go/[email protected]/hook-reader.go:76 +0xbe
io.discard.ReadFrom({}, {0x7f6a69f55180, 0xc0069cedb0})
io/io.go:658 +0x6d
io.copyBuffer({0xe2f5e0, 0x13ba060}, {0x7f6a69f55180, 0xc0069cedb0}, {0x0, 0x0, 0x0})
io/io.go:416 +0x147
io.Copy(...)
io/io.go:389
net/http.(*transferWriter).doBodyCopy(0xc006054a00, {0xe2f5e0?, 0x13ba060?}, {0x7f6a69f55180?, 0xc0069cedb0?})
net/http/transfer.go:412 +0x48
net/http.(*transferWriter).writeBody(0xc006054a00, {0xe2fa20, 0xc003212140})
net/http/transfer.go:375 +0x408
net/http.(*Request).write(0xc006ffdc00, {0xe2fa20, 0xc003212140}, 0x0, 0x0, 0x0)
net/http/request.go:738 +0xbad
net/http.(*persistConn).writeLoop(0xc0039c8a20)
net/http/transport.go:2424 +0x18f
created by net/http.(*Transport).dialConn in goroutine 2172
net/http/transport.go:1777 +0x16f1
Maybe @aead can help a bit since it is sio? Could also be some concurrent access without looking at the code.
I think this could actually be related to the issues we are having with on multipart uploads.
I think this could actually be related to the issues we are having with on multipart uploads.
Which one @klauspost ?
@harshavardhana The one that forced us to turn off checksums on multipart replication or tiering - forget which.
@harshavardhana The one that forced us to turn off checksums on multipart replication or tiering - forget which.
We didn't turn off checksums for that we turned off doing sha256 and md5sum which are expensive.
We still enable crc checksums.