seg fault on app load with big file

[snissn]

Describe the bug Recently added a 20GB file ( piped /dev/urandom into a file for 20gb) and hours later when i returned in the morning it crashed and since then have a segfault when the app loads.

here's the error message

2022-08-05T19:19:43.967Z        ERROR   estuary estuary/replication.go:1613     deal state for deal fd3bb169-2b2a-42cc-aeba-0c2b442275bc from miner f01419959 is error: deal data verification failed: proposal CommP doesn't match calculated CommP        {"app_version": "v0.1.6-34-g9e731d7"}
2022-08-05T19:19:44.015Z        DEBUG   rpc     [email protected]/client.go:542 rpc result      {"type": "*types.TipSet"}
2022-08-05T19:19:44.379Z        DEBUG   rpc     [email protected]/client.go:542 rpc result      {"type": "*types.TipSet"}
2022-08-05T19:19:44.553Z        DEBUG   rpc     [email protected]/client.go:542 rpc result      {"type": "*types.TipSet"}
2022-08-05T19:19:44.603Z        DEBUG   rpc     [email protected]/client.go:542 rpc result      {"type": "*types.Actor"}
2022-08-05T19:19:44.647Z        DEBUG   rpc     [email protected]/client.go:542 rpc result      {"type": "api.MarketBalance"}
2022-08-05T19:19:44.689Z        DEBUG   rpc     [email protected]/client.go:542 rpc result      {"type": "*big.Int"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1ecca91]

goroutine 5399 [running]:
main.(*ContentManager).ensureStorage(_, {_, _}, {0x2, {0x4b46050, 0xeda757e20, 0x98750a0}, {0x4b46050, 0xeda757e20, 0x98750a0}, ...}, ...)
        /usr/local/estuary/replication.go:1425 +0xfb1
main.(*ContentManager).ContentWatcher(0xc005036d00)
        /usr/local/estuary/replication.go:379 +0x46d
created by main.main.func3
        /usr/local/estuary/main.go:659 +0x119c

line 1425 of replication.go:

                        if bl.VerifiedClientBalance.LessThan(big.NewIntUnsigned(uint64(abi.UnpaddedPieceSize(content.Size).Padded()))) {

I could be missing something, I'm not seeing why there's a segfault on this line or even any big memory allocations. pleas let me know if i can provide any additional information like any queries from sql

[en0ma]

@snissn why not replicate it again using same file?

[en0ma]

@snissn closing this, as it has been addressed in https://github.com/application-research/estuary/pull/406

[snissn]

looks right! Thanks! great!