Too many open files when running from bindle

[radu-matei]

When attempting to run a Spin application from a bindle, if there are more static assets than the total number of file handles left on the device, spin up will fail.

This can be worked around by setting ulimit to be higher on macOS, but perhaps Spin should not try to pull all of them in parallel at the same time:

Caused by:
    0: Error fetching remote parcel fermyon.com/0.1.0@f9bc1dc584e3aef90cff417b09ce98d33b9f80fe2a33da805d325639b85fb6f0
    1: Error creating request
    2: error sending request for url (http://localhost:8080/v1/_i/fermyon.com/0.1.0@f9bc1dc584e3aef90cff417b09ce98d33b9f80fe2a33da805d325639b85fb6f0): error trying to connect: tcp open error: Too many open files (os error 24)
    3: error trying to connect: tcp open error: Too many open files (os error 24)
    4: tcp open error: Too many open files (os error 24)
    5: Too many open files (os error 24)

cc @adamreese @itowlson

[lann]

https://docs.rs/futures-util/0.3.21/futures_util/stream/trait.StreamExt.html#method.for_each_concurrent

[radu-matei]

Found the perfect application to test this with — https://github.com/fermyon/spin-kitchensink

Caused by:
    0: Error fetching remote parcel spin-kitchensink/1.0.0@33fcbf51f9315e4415dd2229ec0ec45586054d4f80fa84dce00036f592ea8953
    1: Error creating request
    2: error sending request for url (http://localhost:8080/v1/_i/spin-kitchensink/1.0.0@33fcbf51f9315e4415dd2229ec0ec45586054d4f80fa84dce00036f592ea8953): error trying to connect: tcp connect error: Connection refused (os error 61)
    3: error trying to connect: tcp connect error: Connection refused (os error 61)
    4: tcp connect error: Connection refused (os error 61)
    5: Connection refused (os error 61)    
thread 'main' panicked at 'Cannot prepare component: Error copying assets: 1166 file(s) not copied', /Users/radu/projects/src/github.com/fermyon/spin/crates/loader/src/bindle/mod.rs:81:16
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[dicej]

This may have been fixed with https://github.com/fermyon/spin/pull/440, although I don't have a Mac handy to test with at the moment.

[radu-matei]

It looks like #440 fixed it, thanks!

Closing, but will keep an eye out just in case we missed anything here.

[dicej]

I think this is still an issue, at least on the spin bindle push side of things. On Linux, with ulimit -n set to 1024, I get this when I try to push spin-kitchensink as a bindle:

 $ spin bindle push --file spin.toml
Error: Failed to push bindle to server

Caused by:
    0: Failed to push bindle from '/tmp/.tmplIh9Uw' to server at 'http://127.0.0.1:8000/v1'
    1: Error creating request
    2: error sending request for url (http://127.0.0.1:8000/v1/_i/spin-kitchensink/1.0.0@6394e8e650d04c26ed20b6058d5e93119f544ef1bd00aefb942297d2e2f6c7e1): error trying to connect: tcp open error: Too many open files (os error 24)
    3: error trying to connect: tcp open error: Too many open files (os error 24)
    4: tcp open error: Too many open files (os error 24)
    5: Too many open files (os error 24)

I think this will require a fix in bindle, replacing the use of future::join_all with e.g. StreamExt::buffer_unordered. I tried to do that, but ran into issues with conflicts between tokio_stream::StreamExt and futures::StreamExt. I'm sure it's solvable, but there wasn't a quick fix.

Also, spin still uses future::join_all in several places when preparing and pushing bindles -- at least some of them will need to be replaced with code that limits concurrency to avoid running out of file descriptors.

[itowlson]

Yes, that error's happening within the Bindle StandaloneRead::push function.

[lann]

I'm going to fix up more of these.

[dicej]

The upstream issue is fixed, so I believe the only thing left to do here is upgrade the Bindle dependency in Spin. We could either wait for the next Bindle release or use a Git snapshot until then.

[radu-matei]

It is worth pointing out that we're currently relying on Bindle 0.8.0, and IIRC, there are breaking changes on main that we will have to address — see https://github.com/deislabs/bindle/pull/300