Issues when downloading many files

[ramarty]

I was downloading data for multiple days and the "retrieving" failed a few times. The function still worked, as it's designed to retry. But not sure if there's something going on where it's trying to download too much at one time? Not sure if limiting to, say, 2-3 concurrent downloads would help this issue? (I never seem to get these red bars when downloading just a couple tiles at once).

[ramarty]

My internet was really bad when this happened - when I switched to better internet it didn't happen! But maybe still something to think about?

[Holly-Transport]

I also encountered many instances of file download timing out, like this (I used a smaller country, Jamaica):

[g4brielvs]

I was downloading data for multiple days and the "retrieving" failed a few times. The function still worked, as it's designed to retry. But not sure if there's something going on where it's trying to download too much at one time? Not sure if limiting to, say, 2-3 concurrent downloads would help this issue?

@ramarty @Holly-Transport Thanks (and happy new year)!

That's something to consider. There seems to be 2 things at play. The number of parallel downloads may be too ambitious. In the next release, we can change the default and expose the parameter for the user to choose. Also, especially in slower connections, the default httpx.Client may be timing out too soon. This issue would persist even if we decreased the number of parallel downloads. We can increase the default timeout and also expose it to the user.

All in all, connection issues will always be an upstream constraint imposed by NASA LAADS DAAC, but we can try to find the sweet spot. What do you think?