Runtime Crash: Unexpected fault address fatal error: fault signal SIGBUS: bus error code=0x2

[SealStorage-Jacques]

Checklist

• [X] This is not a question or a support request. If you have any boost related questions, please ask in the discussion forum.
• [X] This is not a new feature request. If it is, please file a feature request instead.
• [X] This is not an enhancement request. If it is, please file a improvement suggestion instead.
• [X] I have searched on the issue tracker and the discussion forum, and there is no existing related issue or discussion.
• [X] I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• [X] I did not make any code changes to boost.

Boost component

• [X] boost daemon - storage providers
• [ ] boost client
• [ ] boost UI
• [ ] boost data-transfer
• [ ] boost index-provider
• [ ] Other

Boost Version

boostd version 1.6.1+git.d31c62f.dirty
go 19.6

Describe the Bug

When importing offline v1.2 deals boostd would randomly crash, we've had several and it crashes in the same Go http routine, see logging section. What also appears to trigger the issue even more often is increasing MaxConcurrentLocalCommp beyond 1.

Logging Information

unexpected fault address 0x7fb392e37f80                                                                                                                                                                 
fatal error: fault                                                                                                                                                                                      
[signal SIGBUS: bus error code=0x2 addr=0x7fb392e37f80 pc=0x7a9e69]                                                                                                                                     
                                                                                                                                                                                                        
goroutine 8521678 [running]:                                                                                                                                                                            
runtime.throw({0x5c2d29a?, 0x2?})                                                                                                                                                                       
        /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0xc000bb3010 sp=0xc000bb2fe0 pc=0x7730fd                                                                                                       
runtime.sigpanic()                                                                                                                                                                                      
        /usr/local/go/src/runtime/signal_unix.go:832 +0xb6 fp=0xc000bb3040 sp=0xc000bb3010 pc=0x78a296                                                                                                  
runtime.memmove()                                                                                                                                                                                       
        /usr/local/go/src/runtime/memmove_amd64.s:339 +0x3c9 fp=0xc000bb3048 sp=0xc000bb3040 pc=0x7a9e69                                                                                                
golang.org/x/exp/mmap.(*ReaderAt).ReadAt(0xc0310d5050, {0xc000bc8000, 0x8000, 0x8000}, 0x72c800000)                                                                                                     
        /usr/local/go/pkg/mod/golang.org/x/[email protected]/mmap/mmap_unix.go:73 +0x2b0 fp=0xc000bb3150 sp=0xc000bb3048 pc=0x2c21610                                              
github.com/ipld/go-car/v2/internal/io.(*offsetReadSeeker).Read(0xc030dc0090, {0xc000bc8000, 0x8000, 0x8000})                                                                                            
        /usr/local/go/pkg/mod/github.com/ipld/go-car/[email protected]/internal/io/offset_read_seeker.go:59 +0x92 fp=0xc000bb31d0 sp=0xc000bb3150 pc=0x2c15bb2                                                  
io.(*LimitedReader).Read(0xc0310d50b0, {0xc000bc8000, 0x8000, 0x8000})
        /usr/local/go/src/io/io.go:477 +0xf3 fp=0xc000bb3238 sp=0xc000bb31d0 pc=0x81c7d3
io.copyBuffer({0x7fdca03c3d50, 0xc026abc350}, {0x6c27620, 0xc0310d50b0}, {0xc000bc8000, 0x8000, 0x8000})
        /usr/local/go/src/io/io.go:427 +0x3ff fp=0xc000bb3380 sp=0xc000bb3238 pc=0x81c31f
io.(*multiReader).writeToWithBuffer(0xc0310d5140, {0x7fdca03c3d50, 0xc026abc350}, {0xc000bc8000, 0x8000, 0x8000})
        /usr/local/go/src/io/multi.go:54 +0x274 fp=0xc000bb3460 sp=0xc000bb3380 pc=0x81e574
io.(*multiReader).WriteTo(0xc0310d5140, {0x7fdca03c3d50, 0xc026abc350})
        /usr/local/go/src/io/multi.go:45 +0xba fp=0xc000bb3500 sp=0xc000bb3460 pc=0x81e25a
io.nopCloserWriterTo.WriteTo({{0x6c276c0, 0xc0310d5140}}, {0x7fdca03c3d50, 0xc026abc350})
        /usr/local/go/src/io/io.go:647 +0xab fp=0xc000bb3588 sp=0xc000bb3500 pc=0x81d90b
io.(*nopCloserWriterTo).WriteTo(0xc026abc2a0, {0x7fdca03c3d50, 0xc026abc350})
        <autogenerated>:1 +0x9b fp=0xc000bb3610 sp=0xc000bb3588 pc=0x82143b
io.copyBuffer({0x7fdca03c3d50, 0xc026abc350}, {0x7fdca03c3d30, 0xc026abc2a0}, {0x0, 0x0, 0x0})
        /usr/local/go/src/io/io.go:409 +0x118 fp=0xc000bb3758 sp=0xc000bb3610 pc=0x81c038
io.Copy({0x7fdca03c3d50, 0xc026abc350}, {0x7fdca03c3d30, 0xc026abc2a0})
        /usr/local/go/src/io/io.go:386 +0x88 fp=0xc000bb37e8 sp=0xc000bb3758 pc=0x81bca8
net/http.(*transferWriter).doBodyCopy(0xc01d32e140, {0x7fdca03c3d50, 0xc026abc350}, {0x7fdca03c3d30, 0xc026abc2a0})
        /usr/local/go/src/net/http/transfer.go:412 +0x6b fp=0xc000bb3850 sp=0xc000bb37e8 pc=0x1073c8b
net/http.(*transferWriter).writeBody(0xc01d32e140, {0x6c27be0, 0xc029b8e060})
        /usr/local/go/src/net/http/transfer.go:359 +0x2da fp=0xc000bb3a88 sp=0xc000bb3850 pc=0x107327a
net/http.(*Request).write(0xc003e88500, {0x6c1ef00, 0xc021ca4140}, 0x0, 0xc01e4f8660, 0x0)
        /usr/local/go/src/net/http/request.go:701 +0x1265 fp=0xc000bb3df8 sp=0xc000bb3a88 pc=0x104a305
net/http.(*persistConn).writeLoop(0xc0107b8000)
        /usr/local/go/src/net/http/transport.go:2395 +0x286 fp=0xc000bb3fc0 sp=0xc000bb3df8 pc=0x108b126
net/http.(*Transport).dialConn.func6()
        /usr/local/go/src/net/http/transport.go:1752 +0x2b fp=0xc000bb3fe0 sp=0xc000bb3fc0 pc=0x1085f2b
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000bb3fe8 sp=0xc000bb3fe0 pc=0x7a8ba1
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1752 +0x2865

Repo Steps

Set MaxConcurrentLocalCommp = 4 in config file Start boostd Import v1.2 offline deals Crash should happen 1 ~ 3 days

[johansealstorage]

Would the 1.7 upgrade with a later version of go perhaps solve this?
We'll give it a test on our side with V1.7 and go v1.20.3