[0.8.1] Fatal error: concurrent map iteration and map write

[Glandos]

This error is triggered with the new version. It was not there before.

mai 21 15:28:16 magneticod-go[11000]: fatal error: concurrent map iteration and map write
mai 21 15:28:16 magneticod-go[11000]: goroutine 23 [running]:
mai 21 15:28:16 magneticod-go[11000]: runtime.throw(0x94ee88, 0x26)
mai 21 15:28:16 magneticod-go[11000]:         /usr/local/go/src/runtime/panic.go:608 +0x72 fp=0x36f9c0c7d70 sp=0x36f9c0c7d40 pc=0x439d12
mai 21 15:28:16 magneticod-go[11000]: runtime.mapiternext(0x36f9c0c7e78)
mai 21 15:28:16 magneticod-go[11000]:         /usr/local/go/src/runtime/map.go:790 +0x525 fp=0x36f9c0c7df8 sp=0x36f9c0c7d70 pc=0x41c8a5
mai 21 15:28:16 magneticod-go[11000]: github.com/boramalper/magnetico/cmd/magneticod/dht/mainline.(*IndexingService).findNeighbors(0x36f9c0a4ae0)
mai 21 15:28:16 magneticod-go[11000]:         /home/bora/labs/magnetico/cmd/magneticod/dht/mainline/indexingService.go:131 +0xf5 fp=0x36f9c0c7ee8 sp=0x36f9c0c7df8 pc=0x795755
mai 21 15:28:16 magneticod-go[11000]: github.com/boramalper/magnetico/cmd/magneticod/dht/mainline.(*IndexingService).index(0x36f9c0a4ae0)
mai 21 15:28:16 magneticod-go[11000]:         /home/bora/labs/magnetico/cmd/magneticod/dht/mainline/indexingService.go:96 +0x2b0 fp=0x36f9c0c7fd8 sp=0x36f9c0c7ee8 pc=0x795010
mai 21 15:28:16 magneticod-go[11000]: runtime.goexit()
mai 21 15:28:16 magneticod-go[11000]:         /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0x36f9c0c7fe0 sp=0x36f9c0c7fd8 pc=0x4699d1
mai 21 15:28:16 magneticod-go[11000]: created by github.com/boramalper/magnetico/cmd/magneticod/dht/mainline.(*IndexingService).Start
mai 21 15:28:16 magneticod-go[11000]:         /home/bora/labs/magnetico/cmd/magneticod/dht/mainline/indexingService.go:78 +0x63
mai 21 15:28:16 magneticod-go[11000]: goroutine 1 [select]:
mai 21 15:28:16 magneticod-go[11000]: main.main()
mai 21 15:28:16 magneticod-go[11000]:         /home/bora/labs/magnetico/cmd/magneticod/main.go:105 +0x74a
mai 21 15:28:16 magneticod-go[11000]: goroutine 19 [syscall]:
mai 21 15:28:16 magneticod-go[11000]: os/signal.signal_recv(0x0)
mai 21 15:28:16 magneticod-go[11000]:         /usr/local/go/src/runtime/sigqueue.go:139 +0x9c
mai 21 15:28:16 magneticod-go[11000]: os/signal.loop()
mai 21 15:28:16 magneticod-go[11000]:         /usr/local/go/src/os/signal/signal_unix.go:23 +0x22
mai 21 15:28:16 magneticod-go[11000]: created by os/signal.init.0
mai 21 15:28:16 magneticod-go[11000]:         /usr/local/go/src/os/signal/signal_unix.go:29 +0x41
mai 21 15:28:16 magneticod-go[11000]: goroutine 34 [select]:
mai 21 15:28:16 magneticod-go[11000]: database/sql.(*DB).connectionOpener(0x36f9c172000, 0x9bc440, 0x36f9c16a000)
mai 21 15:28:16 magneticod-go[11000]:         /usr/local/go/src/database/sql/sql.go:1001 +0xe8
mai 21 15:28:16 magneticod-go[11000]: created by database/sql.OpenDB
mai 21 15:28:16 magneticod-go[11000]:         /usr/local/go/src/database/sql/sql.go:671 +0x15d
mai 21 15:28:16 magneticod-go[11000]: goroutine 35 [select]:
mai 21 15:28:16 magneticod-go[11000]: database/sql.(*DB).connectionResetter(0x36f9c172000, 0x9bc440, 0x36f9c16a000)
mai 21 15:28:16 magneticod-go[11000]:         /usr/local/go/src/database/sql/sql.go:1014 +0xfb
mai 21 15:28:16 magneticod-go[11000]: created by database/sql.OpenDB
mai 21 15:28:16 magneticod-go[11000]:         /usr/local/go/src/database/sql/sql.go:672 +0x193
mai 21 15:28:16 magneticod-go[11000]: goroutine 21 [semacquire]:
mai 21 15:28:16 magneticod-go[11000]: sync.runtime_SemacquireMutex(0x36f9c09eb04, 0x36f9c314600)
mai 21 15:28:16 magneticod-go[11000]:         /usr/local/go/src/runtime/sema.go:71 +0x3d
mai 21 15:28:16 magneticod-go[11000]: sync.(*Mutex).Lock(0x36f9c09eb00)
mai 21 15:28:16 magneticod-go[11000]:         /usr/local/go/src/sync/mutex.go:134 +0xff
mai 21 15:28:16 magneticod-go[11000]: github.com/boramalper/magnetico/cmd/magneticod/dht/mainline.(*IndexingService).onFindNodeResponse(0x36f9c0a4ae0, 0x36f9c1ef800, 0x36f9c1dc030)
mai 21 15:28:16 magneticod-go[11000]:         /home/bora/labs/magnetico/cmd/magneticod/dht/mainline/indexingService.go:145 +0x44
mai 21 15:28:16 magneticod-go[11000]: github.com/boramalper/magnetico/cmd/magneticod/dht/mainline.(*IndexingService).onFindNodeResponse-fm(0x36f9c1ef800, 0x36f9c1dc030)
mai 21 15:28:16 magneticod-go[11000]:         /home/bora/labs/magnetico/cmd/magneticod/dht/mainline/indexingService.go:55 +0x3e
mai 21 15:28:16 magneticod-go[11000]: github.com/boramalper/magnetico/cmd/magneticod/dht/mainline.(*Protocol).onMessage(0x36f9c0a7220, 0x36f9c1ef800, 0x36f9c1dc030)
mai 21 15:28:16 magneticod-go[11000]:         /home/bora/labs/magnetico/cmd/magneticod/dht/mainline/protocol.go:164 +0x624
mai 21 15:28:16 magneticod-go[11000]: github.com/boramalper/magnetico/cmd/magneticod/dht/mainline.(*Protocol).onMessage-fm(0x36f9c1ef800, 0x36f9c1dc030)
mai 21 15:28:16 magneticod-go[11000]:         /home/bora/labs/magnetico/cmd/magneticod/dht/mainline/protocol.go:39 +0x3e
mai 21 15:28:16 magneticod-go[11000]: github.com/boramalper/magnetico/cmd/magneticod/dht/mainline.(*Transport).readMessages(0x36f9c098700)
mai 21 15:28:16 magneticod-go[11000]:         /home/bora/labs/magnetico/cmd/magneticod/dht/mainline/transport.go:120 +0x1b5
mai 21 15:28:16 magneticod-go[11000]: created by github.com/boramalper/magnetico/cmd/magneticod/dht/mainline.(*Transport).Start
mai 21 15:28:16 magneticod-go[11000]:         /home/bora/labs/magnetico/cmd/magneticod/dht/mainline/transport.go:82 +0x15e
mai 21 15:28:16 magneticod-go[11000]: goroutine 22 [chan receive]:
mai 21 15:28:16 magneticod-go[11000]: github.com/boramalper/magnetico/cmd/magneticod/dht/mainline.(*Protocol).updateTokenSecret(0x36f9c0a7220)
mai 21 15:28:16 magneticod-go[11000]:         /home/bora/labs/magnetico/cmd/magneticod/dht/mainline/protocol.go:285 +0x88
mai 21 15:28:16 magneticod-go[11000]: created by github.com/boramalper/magnetico/cmd/magneticod/dht/mainline.(*Protocol).Start
mai 21 15:28:16 magneticod-go[11000]:         /home/bora/labs/magnetico/cmd/magneticod/dht/mainline/protocol.go:59 +0x6a
mai 21 15:28:16 magneticod-go[11000]: goroutine 24 [chan receive]:
mai 21 15:28:16 magneticod-go[11000]: github.com/boramalper/magnetico/cmd/magneticod/bittorrent/metadata.NewSink.func1(0x12a05f200, 0x36f9c0a4b40)
mai 21 15:28:16 magneticod-go[11000]:         /home/bora/labs/magnetico/cmd/magneticod/bittorrent/metadata/sink.go:87 +0xc2
mai 21 15:28:16 magneticod-go[11000]: created by github.com/boramalper/magnetico/cmd/magneticod/bittorrent/metadata.NewSink
mai 21 15:28:16 magneticod-go[11000]:         /home/bora/labs/magnetico/cmd/magneticod/bittorrent/metadata/sink.go:86 +0x127

[kfaulhaber]

This is my first play with magneticod since python, and I am also experiencing the same issue. It runs for a very short period (maybe 1 or 2 hours) and then exits with this same error.

[ranolfi]

I experienced this too, but only once.

[nephaste]

same for me.. on Qnap NAS, but on first start

`[~] # ./magneticod -v 2019-08-11T19:07:37.713+0200 INFO magneticod v0.8.1 has been started. 2019-08-11T19:07:37.713+0200 INFO Copyright (C) 2017-2019 Mert Bora ALPER [email protected]. 2019-08-11T19:07:37.713+0200 INFO Dedicated to Cemile Binay, in whose hands I thrived. 2019-08-11T19:07:37.713+0200 INFO Compiled on 2019-05-21T12:33:32Z 2019-08-11T19:07:37.714+0200 INFO Indexing Service started! 2019-08-11T19:07:38.715+0200 INFO Bootstrapping as routing table is empty... 2019-08-11T19:07:39.715+0200 INFO Latest status: {"n": 2600, "maxNeighbors": 10000} 2019-08-11T19:07:40.715+0200 INFO Latest status: {"n": 3811, "maxNeighbors": 10000} 2019-08-11T19:07:41.715+0200 INFO Latest status: {"n": 3503, "maxNeighbors": 10000} 2019-08-11T19:07:42.715+0200 INFO Sink status {"activeLeeches": 0, "nDeleted": 3, "drainQueue": 0} 2019-08-11T19:07:42.715+0200 INFO Latest status: {"n": 3898, "maxNeighbors": 10000} 2019-08-11T19:07:43.715+0200 INFO Latest status: {"n": 3819, "maxNeighbors": 10000} 2019-08-11T19:07:44.715+0200 INFO Latest status: {"n": 3439, "maxNeighbors": 10000} 2019-08-11T19:07:45.715+0200 INFO Latest status: {"n": 3133, "maxNeighbors": 10000} 2019-08-11T19:07:46.715+0200 INFO Latest status: {"n": 3510, "maxNeighbors": 10000} 2019-08-11T19:07:47.715+0200 INFO Sink status {"activeLeeches": 0, "nDeleted": 1, "drainQueue": 0} 2019-08-11T19:07:47.715+0200 INFO Latest status: {"n": 3364, "maxNeighbors": 10000} 2019-08-11T19:07:48.716+0200 INFO Latest status: {"n": 2976, "maxNeighbors": 10000} 2019-08-11T19:07:49.715+0200 INFO Latest status: {"n": 3352, "maxNeighbors": 10000} 2019-08-11T19:07:50.715+0200 INFO Latest status: {"n": 3797, "maxNeighbors": 10000} 2019-08-11T19:07:51.715+0200 INFO Latest status: {"n": 3312, "maxNeighbors": 10000} 2019-08-11T19:07:52.715+0200 INFO Sink status {"activeLeeches": 0, "nDeleted": 0, "drainQueue": 0} 2019-08-11T19:07:52.715+0200 INFO Latest status: {"n": 3169, "maxNeighbors": 10000} 2019-08-11T19:07:53.715+0200 INFO Latest status: {"n": 3545, "maxNeighbors": 10000} 2019-08-11T19:07:54.715+0200 INFO Latest status: {"n": 3159, "maxNeighbors": 10000} 2019-08-11T19:07:55.716+0200 INFO Latest status: {"n": 3546, "maxNeighbors": 10000} fatal error: concurrent map iteration and map write

goroutine 20 [running]: runtime.throw(0x94ee88, 0x26) /usr/local/go/src/runtime/panic.go:608 +0x72 fp=0xc000052d70 sp=0xc000052d40 pc=0x439d12 runtime.mapiternext(0xc000052e78) /usr/local/go/src/runtime/map.go:790 +0x525 fp=0xc000052df8 sp=0xc000052d70 pc=0x41c8a5 github.com/boramalper/magnetico/cmd/magneticod/dht/mainline.(*IndexingService).findNeighbors(0xc0001441e0) /home/bora/labs/magnetico/cmd/magneticod/dht/mainline/indexingService.go:131 +0xf5 fp=0xc000052ee8 sp=0xc000052df8 pc=0x795755 github.com/boramalper/magnetico/cmd/magneticod/dht/mainline.(*IndexingService).index(0xc0001441e0) /home/bora/labs/magnetico/cmd/magneticod/dht/mainline/indexingService.go:96 +0x2b0 fp=0xc000052fd8 sp=0xc000052ee8 pc=0x795010 runtime.goexit() .........`

[Glandos]

It still happens with 0.9.0

[skobkin]

I'm experiencing it when using my postgres engine implementation on my laptop (Manjaro, Go 1.13.1) every time. But when I'm running it on my desktop (Gentoo, Go 1.13.2) it works fine for 20+ hours and doesn't crash.

Have no idea what difference can cause this problem.

[Musickiller]

It still happens with 0.9.0

It does. Sometimes it runs for some time, finds a few torrents, and sometimes it crashes with this error.

[Musickiller]

somewhat solved be setting --indexer-max-neighbors=1000

[skobkin]

somewhat solved be setting --indexer-max-neighbors=1000

Just reducing the possibility.

[Musickiller]

Just reducing the possibility.

yes, but this way I can at least run the app. It's not a solution, but it's a way to try the app anyway =D

[lruggieri]

Does https://github.com/boramalper/magnetico/pull/225 (already merged) not solve the issue?

[Musickiller]

merged on 9 Nov 2019, while released on 18 Sep 2019. So as it is said, the issue persists in 0.9.0, however I can't say anything about the fix. May be it's time to do a new release?

WARNING:

Do NOT clone the repository to install magnetico, as it is never meant to be stable (except releases of course).

I think that master should be stable. So that you would not need to do releases so that people could check fixes.

[lruggieri]

Maybe, that's up to Boram. But if your purpose, as you stated, is to try the app then try using master for your testing