blackmarblepy
blackmarblepy copied to clipboard
worldbank
Read Operation Timeout
Read Operation Timeout
Description
I tried running the code before and it works perfectly even though it took time to process a whole year's worth of data, hence, I divided the code by month. Now, I tried to rerun it and tried several ways like adjusting the timeout, and the size into a day, but it just usually takes hours until it stops progressing and stays into the same % of "PROCESSING TASKS", is there a way to manage this error? Moreover, the file_skip_if_exists doesn't work since I tried running the same query hoping it will skip the tile with available data, but it just resets the whole process.
Reproducibility
- [ ] The bug is reproducible.
- [x] The bug is intermittent.
- [ ] The bug occurs only under specific conditions.
Screenshots / Error Messages (if applicable)
Here is the screenshot where I ran the same code and it supposedly skipped the file since before it had xKB of size, now it resets to:
Environment
- Operating System: Windows10
@eflplaton Thanks so much for reporting this issue. It seems to be related to known issues #38 and #40. Fortunately, #38 will be addressed in the next release. On #40, please note that BlackMarblePy retrieves the NASA LAADS DAAC - thus it depends on NASA's servers. It will automatically retry failed connections, but after too many requests, you may experience successive errors. For resolution in the next release, I'll try adjusting and reducing the default number of parallel connections. We would greatly appreciate your patience and keep please stay tuned for updates.
Also, if you're looking for monthly composites, I'd suggest using VPN46A3 instead. In fact, we could use both as in this example.
@g4brielvs Makes sense on adjusting number of default number of parallel connections! Wonder if makes sense to have a default, but also add a parameter so users can increase/decrease? For example, if time is less of an issue & want to avoid making too many requests at a time, this would allow setting to parallel connections to 1 in those cases.
Hello, @g4brielvs and @ramarty! I sincerely appreciate your prompt response and the effort you're putting into addressing the issue despite your busy schedule. For the next release addressing issue #38, if it is possible to share, could you kindly give me an estimate on when we might expect the next release? (no pressure intended ☺️)
Thank you once again for your support and understanding.
@eflplaton @ramarty When trying to download from NASA LAADS DAAC directly, I'm encountering connection failures as well - which leads me to believe this is an upstream issue. There is scheduled system reboot, that may or may not be related. Are you running into the same issue?
@eflplaton @ramarty When trying to download from NASA LAADS DAAC directly, I'm encountering connection failures as well - which leads me to believe this is an upstream issue. There is scheduled system reboot, that may or may not be related. Are you running into the same issue?
Hello @g4brielvs, yes I run into the same issue. I even tried VNP46A3 but it's still in 0% after two days.
@eflplaton This could be an issue with the NASA Earthdata token. Could you please check if the token is active?