shell.AddDir fails with "unexpected EOF" on directories with very large files

[rsamf]

When attempting to add a directory with very large files to IPFS, I'm getting an error returned "unexpected EOF". I'm running a private IPFS network with ipfs nodes running ipfs/go-ipfs:release, and cluster nodes running ipfs/ipfs-cluster:v0.14.1.

Tested with go-ipfs-api v0.2.0 and v0.3.0.

To reproduce this while you're in an empty directory, you can create very large files easily like so:

fallocate -l 5120000 a.txt
fallocate -l 5120000 b.txt

Then try running the AddDir method with the path to the directory. With these sizes, AddDir will consistently fail. AddDir can be flaky if the files are lesser in size (say ~5M). It will always work if the files are much lesser (say ~1k).

I noticed that if the directory contains just one gigantic file, AddDir will work without error. But as soon as there is 2 or more files, and at least one of those files is considerably large (at least ~5M), then I will get the "unexpected EOF" error.

[welcome[bot]]

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review. In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment. Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

• "Priority" labels will show how urgent this is for the team.
• "Status" labels will show if this is ready to be worked on, blocked, or in progress.
• "Need" labels will indicate if additional input or analysis is required.

Finally, remember to use https://discuss.ipfs.io if you just need general support.

[rsamf]

Can anyone get to this issue? This is a pretty significant problem

[aschmahmann]

@rsamf can you post a gist or some reproducible code that can be run to reproduce the issue?

You may also want to look at https://github.com/ipfs/go-ipfs-http-client and see if that helps.