Backfilling Trades

[thesmartwon]

When I run: backfiller -trades -from "2018-01-01" -to "2018-01-16" -dir /mnt/e/marketstore/data -apiKey ...

The following error occurs either on the symbol "A" or "AA":

panic: runtime error: makeslice: len out of range

goroutine 115 [running]:
github.com/alpacahq/marketstore/executor.WriteBufferToFileIndirect(0xc001874010, 0xc002da7831, 0x58, 0xcf, 0xc00000000c, 0x0, 0x0)
        /home/thesm/go/src/github.com/alpacahq/marketstore/executor/writer.go:199 +0x6c1
github.com/alpacahq/marketstore/executor.(*WALFileType).writePrimary(0xc00006bc00, 0xc001ae575f, 0x16, 0xc001b58000, 0x1315, 0x1400, 0xc00000001, 0x0, 0x0)
        /home/thesm/go/src/github.com/alpacahq/marketstore/executor/wal.go:337 +0x25b
github.com/alpacahq/marketstore/executor.(*WALFileType).flushToWAL(0xc00006bc00, 0xc00000d980, 0x0, 0x0)
        /home/thesm/go/src/github.com/alpacahq/marketstore/executor/wal.go:300 +0xd3b
github.com/alpacahq/marketstore/executor.(*WALFileType).RequestFlush(0xc00006bc00)
        /home/thesm/go/src/github.com/alpacahq/marketstore/executor/wal.go:863 +0xe4
github.com/alpacahq/marketstore/executor.WriteCSM(0xc00009dd40, 0xc001b06f01, 0x2d, 0xc00009dd70)
        /home/thesm/go/src/github.com/alpacahq/marketstore/executor/writer.go:335 +0x87a
github.com/alpacahq/marketstore/contrib/polygon/backfill.Trades(0xc0019e8790, 0x2, 0x0, 0xed1ef3780, 0x0, 0x0, 0xed1ef3780, 0x0, 0x0, 0xc000124180)
        /home/thesm/go/src/github.com/alpacahq/marketstore/contrib/polygon/backfill/backfill.go:121 +0x628
main.main.func4(0xc0016d5a40, 0xc001350000, 0xc000058000, 0x0, 0xed1ef3780, 0x0)
        /home/thesm/go/src/github.com/alpacahq/marketstore/contrib/polygon/backfill/backfiller/backfiller.go:159 +0xf8
created by main.main
        /home/thesm/go/src/github.com/alpacahq/marketstore/contrib/polygon/backfill/backfiller/backfiller.go:156 +0x6ae

It seems that WriteBufferToFileIndirect is trying to make a slice of size currentRecInfo.Len=5911047525390422 (after having a few hundred legitimate writes) when I printed it. Changing backfill.go:121 to have executor.WriteCSM(csm, false) instead of executor.WriteCSM(csm, true) fixes the issue, but I'm not sure of the deeper implications.

What is causing the writes to act strange, and is this the right fix?

EDIT: I found out that we are trying to write variable-length data instead of fixed-length data when we write trade-level ticks. This makes sense according to the design.