Cannot seed 36GiB file - index too large

[gdluca]

Hi,

is there a file size limit when seeding a file?

│2025-04-02T02:50:24.431761Z ERROR announce: stigmerge_peer::peer::observable: error=failed to encode protocol message: index too large: 2064512 bytes                                                                                        │
│2025-04-02T02:50:24.431826Z  WARN stigmerge_peer::seeder: err="failed to encode protocol message: index too large: 2064512 bytes"                                                                                                            │
│2025-04-02T02:50:27.885894Z ERROR announce: stigmerge_peer::peer::observable: error=failed to encode protocol message: index too large: 2064512 bytes                                                                                        │
│2025-04-02T02:50:27.885961Z  WARN stigmerge_peer::seeder: err="failed to encode protocol message: index too large: 2064512 bytes"

Thanks!

[cmars]

Currently, yes.

The index stores a 256-bit digest for each piece, which in the latest release & main branch, is hard coded to 1MB. Veilid has an upper limit on the total size of the values stored in the DHT, which limits the possible share size.

In #241 (current main focus of development), the piece size will be made configurable when initially seeding the share. This theoretically allows the share size to scale infinitely.

There's probably a practical upper bound to piece size though -- larger pieces mean more wasted bandwidth if the piece fails to validate. So a chaining mechanism to allow indexes to span multiple DHT keys might also be necessary. Chaining would probably work similarly to the haveMap and peerMap references added in that branch, if you're interested in the implementation details.

So... not yet, but hopefully soon!

[gdluca]

I see, thank you for the details. I can manage splitting the file so not really a huge blocker. Unfortunately I can't help with coding due to lack of skill..thanks for working on it!

[cmars]

I'd like to keep this open if you don't mind. Closing it to me means having a way to seed arbitrarily large files, and some guidance in the README.

stigmerge could distribute entire offlike Kiwix sites for example.

[cmars]

With #266 comes the possibility of a configurable piece size -- the protocol will now support it.

[gdluca]

oh nice! Will test next time I have to send something that large :)

[cmars]

The protocol and index structures support it, but I'd need to add CLI options in order for you to make use of it. Still WIP

[gdluca]

Gotcha. When it is ready then! thank you :)