pgfutter dies with a large stacktrace randomly while importing a 150,000 line file

[drewnichols]

I have about 300,000 lines of data to load. pgfutter always dies unless I split the file into 2 and then is dies sometimes. Usually 3 tries and it takes.

Here's the stack trace

failed MSpanList_Insert 0x5e4f70 0x6c6f6e0f0130 0x0
fatal error: MSpanList_Insert

runtime stack:
runtime.MSpanList_Insert(0x578850, 0x5e4f70)
    /goroot/src/runtime/mheap.c:692 +0x8f
runtime.MHeap_Free(0x5787e0, 0x5e4f70, 0x0)
    /goroot/src/runtime/mheap.c:500 +0x5b
runtime.MCentral_FreeSpan(0x57fb38, 0x5e4f70, 0x40, 0xc20809c000, 0xc20809df80, 0x0, 0x64)
    /goroot/src/runtime/mcentral.c:181 +0x1bb
runtime.MSpan_Sweep(0x5e4f70, 0xff00000000, 0x1)
    /goroot/src/runtime/mgc0.c:1099 +0x477
runtime.MHeap_Alloc(0x5787e0, 0x1, 0x1000000000d, 0x15159)
    /goroot/src/runtime/mheap.c:240 +0x66
runtime.MCentral_CacheSpan(0x57ffb8, 0xc2080f8708)
    /goroot/src/runtime/mcentral.c:85 +0x167
runtime.MCache_Refill(0x5e0000, 0xd, 0x10005e4820)
    /goroot/src/runtime/mcache.c:90 +0xa0

goroutine 1 [running]:
runtime.switchtoM()
    /goroot/src/runtime/asm_amd64.s:198 fp=0xc2080f87c8 sp=0xc2080f87c0
runtime.mallocgc(0xc0, 0x0, 0x3, 0x0)
    /goroot/src/runtime/malloc.go:178 +0x849 fp=0xc2080f8878 sp=0xc2080f87c8
runtime.rawstring(0xbe, 0x0, 0x0, 0x0, 0x0, 0x0)
    /goroot/src/runtime/string.go:195 +0x93 fp=0xc2080f88a8 sp=0xc2080f8878
runtime.slicebytetostring(0xc208082480, 0xbe, 0x207, 0x0, 0x0)
    /goroot/src/runtime/string.go:69 +0x4b fp=0xc2080f8938 sp=0xc2080f88a8
encoding/csv.(*Reader).parseRecord(0xc20805c320, 0xc208141580, 0x7, 0x8, 0x0, 0x0)
    /goroot/src/encoding/csv/reader.go:235 +0x449 fp=0xc2080f89f8 sp=0xc2080f8938
encoding/csv.(*Reader).Read(0xc20805c320, 0x0, 0x0, 0x0, 0x0, 0x0)
    /goroot/src/encoding/csv/reader.go:136 +0x5e fp=0xc2080f8a60 sp=0xc2080f89f8
main.copyCSVRows(0xc208033aa0, 0xc20805c320, 0x0, 0xc2080337f5, 0x1, 0xc2080c8100, 0x8, 0x8, 0x0, 0x0, ...)
    /usr/src/pgfutter/csv.go:57 +0xa9 fp=0xc2080f8c08 sp=0xc2080f8a60
main.importCSV(0x7fff5fbffb01, 0x13, 0xc2080f22a0, 0x6b, 0xc208033606, 0x6, 0xc208033630, 0xc, 0x0, 0xc2080f2230, ...)
    /usr/src/pgfutter/csv.go:144 +0x530 fp=0xc2080f8e40 sp=0xc2080f8c08
main.func·003(0xc208080a00)
    /usr/src/pgfutter/pgfutter.go:172 +0x332 fp=0xc2080f8f60 sp=0xc2080f8e40
runtime.call16(0x440268, 0xc2080580b0, 0x800000008)
    /goroot/src/runtime/asm_amd64.s:401 +0x45 fp=0xc2080f8f78 sp=0xc2080f8f60
reflect.Value.call(0x29e880, 0x440268, 0x13, 0x382790, 0x4, 0xc2080f93d0, 0x1, 0x1, 0x0, 0x0, ...)
    /goroot/src/reflect/value.go:419 +0x10e5 fp=0xc2080f9290 sp=0xc2080f8f78
reflect.Value.Call(0x29e880, 0x440268, 0x13, 0xc2080f93d0, 0x1, 0x1, 0x0, 0x0, 0x0)
    /goroot/src/reflect/value.go:296 +0xbc fp=0xc2080f92f0 sp=0xc2080f9290
github.com/codegangsta/cli.HandleAction(0x29e880, 0x440268, 0xc208080a00, 0x0, 0x0)
    /gopath/src/github.com/codegangsta/cli/app.go:487 +0x2b1 fp=0xc2080f9420 sp=0xc2080f92f0
github.com/codegangsta/cli.Command.Run(0x38eb70, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x3dfcf0, 0x18, 0x0, ...)
    /gopath/src/github.com/codegangsta/cli/command.go:191 +0x10a5 fp=0xc2080f9998 sp=0xc2080f9420
github.com/codegangsta/cli.(*App).Run(0xc2080f4000, 0xc20800a000, 0xf, 0xf, 0x0, 0x0)
    /gopath/src/github.com/codegangsta/cli/app.go:240 +0xa76 fp=0xc2080f9c10 sp=0xc2080f9998
main.main()
    /usr/src/pgfutter/pgfutter.go:178 +0xcd3 fp=0xc2080f9f98 sp=0xc2080f9c10
runtime.main()
    /goroot/src/runtime/proc.go:63 +0xf3 fp=0xc2080f9fe0 sp=0xc2080f9f98
runtime.goexit()
    /goroot/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc2080f9fe8 sp=0xc2080f9fe0

goroutine 5 [syscall]:
os/signal.loop()
    /goroot/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
    /goroot/src/os/signal/signal_unix.go:27 +0x35

goroutine 6 [chan receive]:
database/sql.(*DB).connectionOpener(0xc20805d720)
    /goroot/src/database/sql/sql.go:589 +0x4c
created by database/sql.Open
    /goroot/src/database/sql/sql.go:452 +0x31c

goroutine 9 [IO wait]:
net.(*pollDesc).Wait(0xc2080100d0, 0x72, 0x0, 0x0)
    /goroot/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2080100d0, 0x0, 0x0)
    /goroot/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208010070, 0xc2080cc000, 0x1000, 0x1000, 0x0, 0x5f1da0, 0xc208157408)
    /goroot/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc208058030, 0xc2080cc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /goroot/src/net/net.go:121 +0xdc
bufio.(*Reader).fill(0xc2080961e0)
    /goroot/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Read(0xc2080961e0, 0xc208034020, 0x5, 0x200, 0x0, 0x0, 0x0)
    /goroot/src/bufio/bufio.go:174 +0x26c
io.ReadAtLeast(0x5f3398, 0xc2080961e0, 0xc208034020, 0x5, 0x200, 0x5, 0x0, 0x0, 0x0)
    /goroot/src/io/io.go:298 +0xf1
io.ReadFull(0x5f3398, 0xc2080961e0, 0xc208034020, 0x5, 0x200, 0x0, 0x0, 0x0)
    /goroot/src/io/io.go:316 +0x6d
github.com/lib/pq.(*conn).recvMessage(0xc208034000, 0xc208153b00, 0x0, 0x0, 0x0)
    /gopath/src/github.com/lib/pq/conn.go:892 +0x17a
github.com/lib/pq.(*copyin).resploop(0xc20805a3c0)
    /gopath/src/github.com/lib/pq/copy.go:143 +0x55
created by github.com/lib/pq.(*conn).prepareCopyIn
    /gopath/src/github.com/lib/pq/copy.go:90 +0xae5

goroutine 10 [select]:
github.com/cheggaaa/pb.(*ProgressBar).refresher(0xc208080000)
    /gopath/src/github.com/cheggaaa/pb/pb.go:423 +0xff
created by github.com/cheggaaa/pb.(*ProgressBar).Start
    /gopath/src/github.com/cheggaaa/pb/pb.go:123 +0xcd

[lukasmartinelli]

I have about 300,000 lines of data to load. pgfutter always dies unless I split the file into 2 and then is dies sometimes. Usually 3 tries and it takes.

Okay can check. I actually tested this with 0.5 billion lines a year ago but it might be bugs were introduced. This seems like a dependency has changed..like PostgreSQL driver or CSV parser.

Do you have very large columns, with MBs of data in one field?

[drewnichols]

no, most columns are strings and the longest is 418 characters. The total file size is 155 meg.

[alavrard]

I had the same issue with the binary on Mac OS Sierra. I installed the last go version and got pgfutter from the source and it works fine.

[gutenye]

Any fix or reason? I had the same issues on Mac. I imported 90,000 lines, and got MSpanList_Insert error 9/10 times. If you go with larger files, like 100,000 lines, it got 100% error.

UPDATE:

tested on Linux, does not have this problem.

[lukasmartinelli]

Any fix or reason? I had the same issues on Mac. I imported 90,000 lines, and got MSpanList_Insert error 9/10 times. If you go with larger files, like 100,000 lines, it got 100% error.

That's really weird. I've imported 0.5 billion lines with it on Linux.. I can up the Go version and recompile https://github.com/golang/go/issues/17335

[gutenye]

@lukasmartinelli updated, tested on Linux and it works fine. This issue happens at macOS.

[alienxp03]

Let me know if you need any more info with this issue. Can confirm it only happened on Mac. I'm having millions rows in one CSV, so it's pretty unusable on Mac. Had to use Linux to test it out

[mhart]

Just confirmed – same issue, almost exact same stacktrace – line-delimited JSON file, 48k lines, Mac OS X 10.12.2, PostgreSQL 9.6.2

Got to 39.81 MB / 135.65 MB before it failed

[chriszrc]

Stack trace is a little different here, but similar case, macos sierra, postgres 9.6, ~400mb csv, fails about ~180mb with:

fatal error: unexpected signal during runtime execution [signal 0xb code=0x1 addr=0xb01dfacedebac1e pc=0x35e99]

runtime stack: runtime.gothrow(0x40e3f0, 0x2a) /goroot/src/runtime/panic.go:503 +0x8e fp=0xc20802bcd8 sp=0xc20802bcc0 runtime.sigpanic() /goroot/src/runtime/sigpanic_unix.go:14 +0x5e fp=0xc20802bd28 sp=0xc20802bcd8 runtime.MSpanList_Remove(0x5e69d8) /goroot/src/runtime/mheap.c:676 +0x29 fp=0xc20802bd30 sp=0xc20802bd28 runtime.MCentral_UncacheSpan(0x57f358, 0x5e69d8) /goroot/src/runtime/mcentral.c:121 +0x7f fp=0xc20802bd50 sp=0xc20802bd30 runtime.MCache_ReleaseAll(0x5e0000) /goroot/src/runtime/mcache.c:111 +0x6a fp=0xc20802bd78 sp=0xc20802bd50 flushallmcaches() /goroot/src/runtime/mgc0.c:1224 +0x61 fp=0xc20802bda0 sp=0xc20802bd78 markroot(0xc20800a070, 0xc200000004) /goroot/src/runtime/mgc0.c:536 +0x20a fp=0xc20802be00 sp=0xc20802bda0 runtime.parfordo(0xc20800a070) /goroot/src/runtime/parfor.c:76 +0xb2 fp=0xc20802be80 sp=0xc20802be00 gc(0xc20802bfb8) /goroot/src/runtime/mgc0.c:1442 +0x25e fp=0xc20802bf98 sp=0xc20802be80 runtime.gc_m() /goroot/src/runtime/mgc0.c:1371 +0xe0 fp=0xc20802bfd0 sp=0xc20802bf98 runtime.onM(0xc208010000) /goroot/src/runtime/asm_amd64.s:257 +0x68 fp=0xc20802bfd8 sp=0xc20802bfd0 runtime.mstart() /goroot/src/runtime/proc.c:818 fp=0xc20802bfe0 sp=0xc20802bfd8

goroutine 1 [garbage collection]: runtime.switchtoM() /goroot/src/runtime/asm_amd64.s:198 fp=0xc2080f4790 sp=0xc2080f4788 runtime.gogc(0x0) /goroot/src/runtime/malloc.go:469 +0x1cf fp=0xc2080f47c8 sp=0xc2080f4790 runtime.mallocgc(0x10, 0x0, 0x3, 0x0) /goroot/src/runtime/malloc.go:341 +0x391 fp=0xc2080f4878 sp=0xc2080f47c8 runtime.rawstring(0xd, 0x0, 0x0, 0x0, 0x0, 0x0) /goroot/src/runtime/string.go:195 +0x93 fp=0xc2080f48a8 sp=0xc2080f4878 runtime.slicebytetostring(0xc208038554, 0xd, 0x40, 0x0, 0x0) /goroot/src/runtime/string.go:69 +0x4b fp=0xc2080f4938 sp=0xc2080f48a8 encoding/csv.(*Reader).parseRecord(0xc208038500, 0xc208165ff0, 0x1, 0x1, 0x0, 0x0) /goroot/src/encoding/csv/reader.go:235 +0x449 fp=0xc2080f49f8 sp=0xc2080f4938 encoding/csv.(*Reader).Read(0xc208038500, 0x0, 0x0, 0x0, 0x0, 0x0) /goroot/src/encoding/csv/reader.go:136 +0x5e fp=0xc2080f4a60 sp=0xc2080f49f8 main.copyCSVRows(0xc208031060, 0xc208038500, 0x0, 0xc208031735, 0x1, 0xc2080c6100, 0x5, 0x8, 0x0, 0x0, ...) /usr/src/pgfutter/csv.go:57 +0xa9 fp=0xc2080f4c08 sp=0xc2080f4a60 main.importCSV(0x7fff5fbffb7e, 0x4d, 0xc2080f02a0, 0x65, 0xc208031556, 0x6, 0x7fff5fbffbaf, 0x18, 0x0, 0x571590, ...) /usr/src/pgfutter/csv.go:144 +0x530 fp=0xc2080f4e40 sp=0xc2080f4c08 main.func·003(0xc208058a00) /usr/src/pgfutter/pgfutter.go:172 +0x332 fp=0xc2080f4f60 sp=0xc2080f4e40 runtime.call16(0x440268, 0xc2080340b0, 0x800000008) /goroot/src/runtime/asm_amd64.s:401 +0x45 fp=0xc2080f4f78 sp=0xc2080f4f60 reflect.Value.call(0x29e880, 0x440268, 0x13, 0x382790, 0x4, 0xc2080f53d0, 0x1, 0x1, 0x0, 0x0, ...) /goroot/src/reflect/value.go:419 +0x10e5 fp=0xc2080f5290 sp=0xc2080f4f78 reflect.Value.Call(0x29e880, 0x440268, 0x13, 0xc2080f53d0, 0x1, 0x1, 0x0, 0x0, 0x0) /goroot/src/reflect/value.go:296 +0xbc fp=0xc2080f52f0 sp=0xc2080f5290 github.com/codegangsta/cli.HandleAction(0x29e880, 0x440268, 0xc208058a00, 0x0, 0x0) /gopath/src/github.com/codegangsta/cli/app.go:487 +0x2b1 fp=0xc2080f5420 sp=0xc2080f52f0 github.com/codegangsta/cli.Command.Run(0x38eb70, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x3dfcf0, 0x18, 0x0, ...) /gopath/src/github.com/codegangsta/cli/command.go:191 +0x10a5 fp=0xc2080f5998 sp=0xc2080f5420 github.com/codegangsta/cli.(*App).Run(0xc20800c180, 0xc20800a000, 0x7, 0x7, 0x0, 0x0) /gopath/src/github.com/codegangsta/cli/app.go:240 +0xa76 fp=0xc2080f5c10 sp=0xc2080f5998 main.main() /usr/src/pgfutter/pgfutter.go:178 +0xcd3 fp=0xc2080f5f98 sp=0xc2080f5c10 runtime.main() /goroot/src/runtime/proc.go:63 +0xf3 fp=0xc2080f5fe0 sp=0xc2080f5f98 runtime.goexit() /goroot/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc2080f5fe8 sp=0xc2080f5fe0

goroutine 2 [force gc (idle)]: runtime.gopark(0x3b610, 0x568bc0, 0x3b1530, 0xf) /goroot/src/runtime/proc.go:130 +0x105 fp=0xc208018798 sp=0xc208018768 runtime.goparkunlock(0x568bc0, 0x3b1530, 0xf) /goroot/src/runtime/proc.go:136 +0x48 fp=0xc2080187c0 sp=0xc208018798 runtime.forcegchelper() /goroot/src/runtime/proc.go:99 +0xce fp=0xc2080187e0 sp=0xc2080187c0 runtime.goexit() /goroot/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc2080187e8 sp=0xc2080187e0 created by runtime.init·4 /goroot/src/runtime/proc.go:87 +0x25

goroutine 3 [runnable]: runtime.Gosched() /goroot/src/runtime/proc.go:113 +0x14 fp=0xc20801bfc0 sp=0xc20801bfb0 runtime.bgsweep() /goroot/src/runtime/mgc0.go:87 +0x59 fp=0xc20801bfe0 sp=0xc20801bfc0 runtime.goexit() /goroot/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc20801bfe8 sp=0xc20801bfe0 created by gc /goroot/src/runtime/mgc0.c:1386

goroutine 4 [finalizer wait]: runtime.gopark(0x3b610, 0x571518, 0x3b0e70, 0xe) /goroot/src/runtime/proc.go:130 +0x105 fp=0xc208016f30 sp=0xc208016f00 runtime.goparkunlock(0x571518, 0x3b0e70, 0xe) /goroot/src/runtime/proc.go:136 +0x48 fp=0xc208016f58 sp=0xc208016f30 runtime.runfinq() /goroot/src/runtime/malloc.go:727 +0xba fp=0xc208016fe0 sp=0xc208016f58 runtime.goexit() /goroot/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc208016fe8 sp=0xc208016fe0 created by runtime.createfing /goroot/src/runtime/malloc.go:707 +0x5e

goroutine 5 [syscall]: runtime.notetsleepg(0x571d20, 0xffffffffffffffff, 0x569901) /goroot/src/runtime/lock_sema.go:266 +0x8c fp=0xc20801b768 sp=0xc20801b728 runtime.signal_recv(0x0) /goroot/src/runtime/sigqueue.go:109 +0x135 fp=0xc20801b7a0 sp=0xc20801b768 os/signal.loop() /goroot/src/os/signal/signal_unix.go:21 +0x1f fp=0xc20801b7e0 sp=0xc20801b7a0 runtime.goexit() /goroot/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc20801b7e8 sp=0xc20801b7e0 created by os/signal.init·1 /goroot/src/os/signal/signal_unix.go:27 +0x35

goroutine 6 [chan receive]: runtime.gopark(0x3b610, 0xc2080806b8, 0x3aded0, 0xc) /goroot/src/runtime/proc.go:130 +0x105 fp=0xc2080166b0 sp=0xc208016680 runtime.goparkunlock(0xc2080806b8, 0x3aded0, 0xc) /goroot/src/runtime/proc.go:136 +0x48 fp=0xc2080166d8 sp=0xc2080166b0 runtime.chanrecv(0x28c540, 0xc208080660, 0xc2080167c8, 0x1, 0x0) /goroot/src/runtime/chan.go:467 +0x82c fp=0xc208016778 sp=0xc2080166d8 runtime.chanrecv2(0x28c540, 0xc208080660, 0xc2080167c8, 0x0) /goroot/src/runtime/chan.go:316 +0x2b fp=0xc2080167a8 sp=0xc208016778 database/sql.(*DB).connectionOpener(0xc208039720) /goroot/src/database/sql/sql.go:589 +0x4c fp=0xc2080167d8 sp=0xc2080167a8 runtime.goexit() /goroot/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc2080167e0 sp=0xc2080167d8 created by database/sql.Open /goroot/src/database/sql/sql.go:452 +0x31c

goroutine 7 [syscall]: runtime.notetsleepg(0x568dd8, 0xbebb75d, 0x16) /goroot/src/runtime/lock_sema.go:266 +0x8c fp=0xc208017768 sp=0xc208017728 runtime.timerproc() /goroot/src/runtime/time.go:207 +0xfa fp=0xc2080177e0 sp=0xc208017768 runtime.goexit() /goroot/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc2080177e8 sp=0xc2080177e0 created by runtime.addtimerLocked /goroot/src/runtime/time.go:113 +0x1ba

goroutine 9 [IO wait]: runtime.gopark(0x1a080, 0x5f3230, 0x384e50, 0x7) /goroot/src/runtime/proc.go:130 +0x105 fp=0xc2080152e0 sp=0xc2080152b0 runtime.netpollblock(0x5f3208, 0x72, 0xc208111a38) /goroot/src/runtime/netpoll.go:347 +0x170 fp=0xc208015320 sp=0xc2080152e0 runtime.netpollWait(0x5f3208, 0x72, 0x8) /goroot/src/runtime/netpoll.go:150 +0x68 fp=0xc208015340 sp=0xc208015320 net.(*pollDesc).Wait(0xc2080f0060, 0x72, 0x0, 0x0) /goroot/src/net/fd_poll_runtime.go:84 +0x47 fp=0xc208015360 sp=0xc208015340 net.(*pollDesc).WaitRead(0xc2080f0060, 0x0, 0x0) /goroot/src/net/fd_poll_runtime.go:89 +0x43 fp=0xc208015388 sp=0xc208015360 net.(*netFD).Read(0xc2080f0000, 0xc2080ca000, 0x1000, 0x1000, 0x0, 0x5f1d90, 0xc208111a30) /goroot/src/net/fd_unix.go:242 +0x40f fp=0xc208015428 sp=0xc208015388 net.(*conn).Read(0xc208034018, 0xc2080ca000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /goroot/src/net/net.go:121 +0xdc fp=0xc208015470 sp=0xc208015428 bufio.(*Reader).fill(0xc208080060) /goroot/src/bufio/bufio.go:97 +0x1ce fp=0xc208015518 sp=0xc208015470 bufio.(*Reader).Read(0xc208080060, 0xc208056f20, 0x5, 0x200, 0x0, 0x0, 0x0) /goroot/src/bufio/bufio.go:174 +0x26c fp=0xc2080155b0 sp=0xc208015518 io.ReadAtLeast(0x5f32f0, 0xc208080060, 0xc208056f20, 0x5, 0x200, 0x5, 0x0, 0x0, 0x0) /goroot/src/io/io.go:298 +0xf1 fp=0xc208015608 sp=0xc2080155b0 io.ReadFull(0x5f32f0, 0xc208080060, 0xc208056f20, 0x5, 0x200, 0x0, 0x0, 0x0) /goroot/src/io/io.go:316 +0x6d fp=0xc208015658 sp=0xc208015608 github.com/lib/pq.(*conn).recvMessage(0xc208056f00, 0xc208112e40, 0x0, 0x0, 0x0) /gopath/src/github.com/lib/pq/conn.go:892 +0x17a fp=0xc208015728 sp=0xc208015658 github.com/lib/pq.(*copyin).resploop(0xc208036460) /gopath/src/github.com/lib/pq/copy.go:143 +0x55 fp=0xc2080157d8 sp=0xc208015728 runtime.goexit() /goroot/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc2080157e0 sp=0xc2080157d8 created by github.com/lib/pq.(*conn).prepareCopyIn /gopath/src/github.com/lib/pq/copy.go:90 +0xae5

goroutine 10 [select]: runtime.gopark(0x1ebd0, 0xc208015f40, 0x39afb0, 0x6) /goroot/src/runtime/proc.go:130 +0x105 fp=0xc208015d58 sp=0xc208015d28 runtime.selectgoImpl(0xc208015f40, 0x0, 0x18) /goroot/src/runtime/select.go:366 +0xb2c fp=0xc208015ef0 sp=0xc208015d58 runtime.selectgo(0xc208015f40) /goroot/src/runtime/select.go:183 +0x12 fp=0xc208015f10 sp=0xc208015ef0 github.com/cheggaaa/pb.(*ProgressBar).refresher(0xc208058000) /gopath/src/github.com/cheggaaa/pb/pb.go:423 +0xff fp=0xc208015fd8 sp=0xc208015f10 runtime.goexit() /goroot/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc208015fe0 sp=0xc208015fd8 created by github.com/cheggaaa/pb.(*ProgressBar).Start /gopath/src/github.com/cheggaaa/pb/pb.go:123 +0xcd

[kaustuv]

macOS Sierra 10.12.3, postgres 9.6.2, tab delimited file, breaks at 400mb of a 18gb file. below is the stacktrace:

failed MSpanList_Insert 0x5e6838 0x1bb49d3b774d 0x0 fatal error: MSpanList_Insert

runtime stack: runtime.MSpanList_Insert(0x578850, 0x5e6838) /goroot/src/runtime/mheap.c:692 +0x8f runtime.MHeap_Free(0x5787e0, 0x5e6838, 0x0) /goroot/src/runtime/mheap.c:500 +0x5b runtime.MCentral_FreeSpan(0x5808b8, 0x5e6838, 0x15, 0xc208168000, 0xc208169e00, 0x0, 0x64) /goroot/src/runtime/mcentral.c:181 +0x1bb runtime.MSpan_Sweep(0x5e6838, 0x4bc300004b00, 0x1) /goroot/src/runtime/mgc0.c:1099 +0x477 runtime.MHeap_Alloc(0x5787e0, 0x1, 0x10000000003, 0x15159) /goroot/src/runtime/mheap.c:240 +0x66 runtime.MCentral_CacheSpan(0x57f478, 0xc2080ee940) /goroot/src/runtime/mcentral.c:85 +0x167 runtime.MCache_Refill(0x5e0000, 0x3, 0x8005e6698) /goroot/src/runtime/mcache.c:90 +0xa0

goroutine 1 [running]: runtime.switchtoM() /goroot/src/runtime/asm_amd64.s:198 fp=0xc2080ee738 sp=0xc2080ee730 runtime.mallocgc(0x20, 0x344940, 0x100000000, 0x1) /goroot/src/runtime/malloc.go:178 +0x849 fp=0xc2080ee7e8 sp=0xc2080ee738 runtime.newobject(0x344940, 0xc2080a42d0) /goroot/src/runtime/malloc.go:353 +0x49 fp=0xc2080ee810 sp=0xc2080ee7e8 database/sql.resultFromStatement(0x5f33d0, 0xc208092660, 0x5f35b0, 0xc208038230, 0xc208161e00, 0x18, 0x18, 0x0, 0x0, 0x10, ...) /goroot/src/database/sql/sql.go:1324 +0x35 fp=0xc2080ee940 sp=0xc2080ee810 database/sql.(*Stmt).Exec(0xc2080a4280, 0xc208161e00, 0x18, 0x18, 0x0, 0x0, 0x0, 0x0) /goroot/src/database/sql/sql.go:1315 +0x2b0 fp=0xc2080eea18 sp=0xc2080ee940 main.(*Import).AddRow(0xc20810a970, 0xc208161e00, 0x18, 0x18, 0x0, 0x0) /usr/src/pgfutter/import.go:60 +0x5c fp=0xc2080eea60 sp=0xc2080eea18 main.copyCSVRows(0xc20810a970, 0xc20803a460, 0x0, 0xc208032370, 0x1, 0xc2080e8180, 0x18, 0x18, 0x0, 0x0, ...) /usr/src/pgfutter/csv.go:82 +0x5a2 fp=0xc2080eec08 sp=0xc2080eea60 main.importCSV(0x7fff5fbff934, 0x21, 0xc2080e6000, 0x65, 0xc2080335f6, 0x6, 0xc208033890, 0xa, 0x0, 0xc20804a000, ...) /usr/src/pgfutter/csv.go:144 +0x530 fp=0xc2080eee40 sp=0xc2080eec08 main.func·003(0xc20804aa00) /usr/src/pgfutter/pgfutter.go:172 +0x332 fp=0xc2080eef60 sp=0xc2080eee40 runtime.call16(0x440268, 0xc2080360b0, 0x800000008) /goroot/src/runtime/asm_amd64.s:401 +0x45 fp=0xc2080eef78 sp=0xc2080eef60 reflect.Value.call(0x29e880, 0x440268, 0x13, 0x382790, 0x4, 0xc2080ef3d0, 0x1, 0x1, 0x0, 0x0, ...) /goroot/src/reflect/value.go:419 +0x10e5 fp=0xc2080ef290 sp=0xc2080eef78 reflect.Value.Call(0x29e880, 0x440268, 0x13, 0xc2080ef3d0, 0x1, 0x1, 0x0, 0x0, 0x0) /goroot/src/reflect/value.go:296 +0xbc fp=0xc2080ef2f0 sp=0xc2080ef290 github.com/codegangsta/cli.HandleAction(0x29e880, 0x440268, 0xc20804aa00, 0x0, 0x0) /gopath/src/github.com/codegangsta/cli/app.go:487 +0x2b1 fp=0xc2080ef420 sp=0xc2080ef2f0 github.com/codegangsta/cli.Command.Run(0x38eb70, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x3dfcf0, 0x18, 0x0, ...) /gopath/src/github.com/codegangsta/cli/command.go:191 +0x10a5 fp=0xc2080ef998 sp=0xc2080ef420 github.com/codegangsta/cli.(*App).Run(0xc2080e8000, 0xc20800a000, 0xf, 0xf, 0x0, 0x0) /gopath/src/github.com/codegangsta/cli/app.go:240 +0xa76 fp=0xc2080efc10 sp=0xc2080ef998 main.main() /usr/src/pgfutter/pgfutter.go:178 +0xcd3 fp=0xc2080eff98 sp=0xc2080efc10 runtime.main() /goroot/src/runtime/proc.go:63 +0xf3 fp=0xc2080effe0 sp=0xc2080eff98 runtime.goexit() /goroot/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc2080effe8 sp=0xc2080effe0

goroutine 5 [syscall]: os/signal.loop() /goroot/src/os/signal/signal_unix.go:21 +0x1f created by os/signal.init·1 /goroot/src/os/signal/signal_unix.go:27 +0x35

goroutine 6 [chan receive]: database/sql.(*DB).connectionOpener(0xc20803a000) /goroot/src/database/sql/sql.go:589 +0x4c created by database/sql.Open /goroot/src/database/sql/sql.go:452 +0x31c

goroutine 9 [IO wait]: net.(*pollDesc).Wait(0xc2080e6140, 0x72, 0x0, 0x0) /goroot/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2080e6140, 0x0, 0x0) /goroot/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2080e60e0, 0xc2080ca000, 0x1000, 0x1000, 0x0, 0x5f1cc0, 0xc2081511b0) /goroot/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc208036038, 0xc2080ca000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /goroot/src/net/net.go:121 +0xdc bufio.(*Reader).fill(0xc2080921e0) /goroot/src/bufio/bufio.go:97 +0x1ce bufio.(*Reader).Read(0xc2080921e0, 0xc208078020, 0x5, 0x200, 0x0, 0x0, 0x0) /goroot/src/bufio/bufio.go:174 +0x26c io.ReadAtLeast(0x5f31f8, 0xc2080921e0, 0xc208078020, 0x5, 0x200, 0x5, 0x0, 0x0, 0x0) /goroot/src/io/io.go:298 +0xf1 io.ReadFull(0x5f31f8, 0xc2080921e0, 0xc208078020, 0x5, 0x200, 0x0, 0x0, 0x0) /goroot/src/io/io.go:316 +0x6d github.com/lib/pq.(*conn).recvMessage(0xc208078000, 0xc20801ec80, 0x0, 0x0, 0x0) /gopath/src/github.com/lib/pq/conn.go:892 +0x17a github.com/lib/pq.(*copyin).resploop(0xc208038230) /gopath/src/github.com/lib/pq/copy.go:143 +0x55 created by github.com/lib/pq.(*conn).prepareCopyIn /gopath/src/github.com/lib/pq/copy.go:90 +0xae5

goroutine 10 [select]: github.com/cheggaaa/pb.(*ProgressBar).refresher(0xc20804a140) /gopath/src/github.com/cheggaaa/pb/pb.go:423 +0xff created by github.com/cheggaaa/pb.(*ProgressBar).Start /gopath/src/github.com/cheggaaa/pb/pb.go:123 +0xcd

[kps143]

Issue still occurring. Any news on a fix?

failed MSpanList_Insert 0x5d2c70 0x603777e3daa 0x0 fatal error: MSpanList_Insert

runtime stack: runtime.MSpanList_Insert(0x567770, 0x5d2c70) /goroot/src/runtime/mheap.c:692 +0x8f runtime.MHeap_Free(0x567700, 0x5d2c70, 0x0) /goroot/src/runtime/mheap.c:500 +0x5b runtime.MCentral_FreeSpan(0x56e278, 0x5d2c70, 0x200, 0xc20811e000, 0xc20811fff0, 0x0, 0x64) /goroot/src/runtime/mcentral.c:181 +0x1bb runtime.MSpan_Sweep(0x5d2c70, 0x182500001800, 0x1) /goroot/src/runtime/mgc0.c:1099 +0x477 runtime.MHeap_Alloc(0x567700, 0x1, 0x10000000005, 0x14e09) /goroot/src/runtime/mheap.c:240 +0x66 runtime.MCentral_CacheSpan(0x56e5d8, 0x5d2520) /goroot/src/runtime/mcentral.c:85 +0x167 runtime.MCache_Refill(0x5cd000, 0x5, 0x5d2520) /goroot/src/runtime/mcache.c:90 +0xa0

goroutine 1 [running]: runtime.switchtoM() /goroot/src/runtime/asm_amd64.s:198 fp=0xc2080f2788 sp=0xc2080f2780 runtime.mallocgc(0x40, 0x0, 0x3, 0x0) /goroot/src/runtime/malloc.go:178 +0x849 fp=0xc2080f2838 sp=0xc2080f2788 runtime.rawstring(0x32, 0x0, 0x0, 0x0, 0x0, 0x0) /goroot/src/runtime/string.go:195 +0x93 fp=0xc2080f2868 sp=0xc2080f2838 runtime.slicebytetostring(0xc2080fe000, 0x32, 0x81, 0x0, 0x0) /goroot/src/runtime/string.go:69 +0x4b fp=0xc2080f28f8 sp=0xc2080f2868 encoding/csv.(*Reader).parseRecord(0xc208038460, 0xc208166400, 0x25, 0x40, 0x0, 0x0) /goroot/src/encoding/csv/reader.go:235 +0x449 fp=0xc2080f29b8 sp=0xc2080f28f8 encoding/csv.(*Reader).Read(0xc208038460, 0x0, 0x0, 0x0, 0x0, 0x0) /goroot/src/encoding/csv/reader.go:136 +0x5e fp=0xc2080f2a20 sp=0xc2080f29b8 main.copyCSVRows(0xc20810ba30, 0xc208038460, 0x0, 0xc208033655, 0x1, 0xc2080ae000, 0x2a, 0x40, 0x0, 0x0, ...) /usr/src/pgfutter/csv.go:57 +0xa9 fp=0xc2080f2bc8 sp=0xc2080f2a20 main.importCSV(0x7fff5fbffb3a, 0xc, 0xc2080ee230, 0x64, 0xc208033426, 0x6, 0xc208033644, 0x9, 0x0, 0x5604b8, ...) /usr/src/pgfutter/csv.go:144 +0x530 fp=0xc2080f2e00 sp=0xc2080f2bc8 main.func·003(0xc20807c8c0) /usr/src/pgfutter/pgfutter.go:157 +0x332 fp=0xc2080f2f20 sp=0xc2080f2e00 github.com/codegangsta/cli.Command.Run(0x382d50, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x3d31b0, 0x18, 0x0, ...) /gopath/src/github.com/codegangsta/cli/command.go:179 +0x1403 fp=0xc2080f3480 sp=0xc2080f2f20 github.com/codegangsta/cli.(*App).Run(0xc20800c420, 0xc20800a000, 0x5, 0x5, 0x0, 0x0) /gopath/src/github.com/codegangsta/cli/app.go:196 +0x14b5 fp=0xc2080f3cc0 sp=0xc2080f3480 main.main() /usr/src/pgfutter/pgfutter.go:163 +0xba4 fp=0xc2080f3f98 sp=0xc2080f3cc0 runtime.main() /goroot/src/runtime/proc.go:63 +0xf3 fp=0xc2080f3fe0 sp=0xc2080f3f98 runtime.goexit() /goroot/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc2080f3fe8 sp=0xc2080f3fe0

goroutine 5 [syscall]: os/signal.loop() /goroot/src/os/signal/signal_unix.go:21 +0x1f created by os/signal.init·1 /goroot/src/os/signal/signal_unix.go:27 +0x35

goroutine 6 [chan receive]: database/sql.(*DB).connectionOpener(0xc208039860) /goroot/src/database/sql/sql.go:589 +0x4c created by database/sql.Open /goroot/src/database/sql/sql.go:452 +0x31c

goroutine 9 [IO wait]: net.(*pollDesc).Wait(0xc2080ee060, 0x72, 0x0, 0x0) /goroot/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2080ee060, 0x0, 0x0) /goroot/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2080ee000, 0xc2080d0000, 0x1000, 0x1000, 0x0, 0x5decc0, 0xc20813ae18) /goroot/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc208036018, 0xc2080d0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /goroot/src/net/net.go:121 +0xdc bufio.(*Reader).fill(0xc208098060) /goroot/src/bufio/bufio.go:97 +0x1ce bufio.(*Reader).Read(0xc208098060, 0xc208034f20, 0x5, 0x200, 0x0, 0x0, 0x0) /goroot/src/bufio/bufio.go:174 +0x26c io.ReadAtLeast(0x5e01f8, 0xc208098060, 0xc208034f20, 0x5, 0x200, 0x5, 0x0, 0x0, 0x0) /goroot/src/io/io.go:298 +0xf1 io.ReadFull(0x5e01f8, 0xc208098060, 0xc208034f20, 0x5, 0x200, 0x0, 0x0, 0x0) /goroot/src/io/io.go:316 +0x6d github.com/lib/pq.(*conn).recvMessage(0xc208034f00, 0xc20811c740, 0x0, 0x0, 0x0) /gopath/src/github.com/lib/pq/conn.go:892 +0x17a github.com/lib/pq.(*copyin).resploop(0xc20800a370) /gopath/src/github.com/lib/pq/copy.go:143 +0x55 created by github.com/lib/pq.(*conn).prepareCopyIn /gopath/src/github.com/lib/pq/copy.go:90 +0xae5

goroutine 10 [select]: github.com/cheggaaa/pb.(*ProgressBar).refresher(0xc20807c000) /gopath/src/github.com/cheggaaa/pb/pb.go:410 +0xff created by github.com/cheggaaa/pb.(*ProgressBar).Start /gopath/src/github.com/cheggaaa/pb/pb.go:121 +0xc7

[seamusabshere]

yep, still happening for me too

[ludo]

With the provided binary package for OSX I also ran into the issue of random failures while importing a JSON file into PostgreSQL (fatal error: MSpanList_Insert) on macOS Sierra 10.12.5 with PostgreSQL 9.6.3.

However, when building the binary from source it works. Steps:

brew install go
go get github.com/lukasmartinelli/pgfutter
~/go/bin/pgfutter json data.json

[geastham]

Can confirm @ludo's fix works for me.

MacOS Sierra 10.12.5 PostgreSQL 9.6.3

[dacoinminster]

This crash now happens even when trying to run pgfutter --help (see issue #48)

Installing from source as described above fixes the problem.

Suggest updating the OSX installation instructions in the readme.