factbook GitHub Actions workflow fails (HTTP 404) on downloading ek.json

Hi @geraldb

as the title of this issue reads. The update GitHub Actions workflow fails on ek.json

==> [55/260] ek Equatorial Guinea / Africa -- Countries:
  sleep 0.5 sec(s)...
GET https://www.cia.gov/the-world-factbook/geos/ek.json...
Error: Process completed with exit code 1.

The url https://www.cia.gov/the-world-factbook/geos/ek.json throws the error code 404.

I'll add further insights on this in the comments below.

Regards, Jean-Luc

Mar 13 '22 23:03 eUgEntOptIc44

The html page for the country at https://www.cia.gov/the-world-factbook/countries/equatorial-guinea/ as well as the json content of Germany are still accessible. It might be just a human error.

However integrating execption handling per url in the code of the factbook script might be a good idea. As at the time of writing (to my personal understanding) the occurrence of an error in the parsing of one json url aborts the whole script.

Mar 13 '22 23:03 eUgEntOptIc44

I'd be very grateful if @geraldb could look into this any time soon. Please do keep up the great work of keeping this project up and running!

Mar 13 '22 23:03 eUgEntOptIc44

Sorry looks like the situation changed faster than expected. I just managed to successfully call the previously faulty url. Maybe somebody at the CIA has starred the repo. You never know... 😉

But I'd like to highlight that (more) exception handling would be nice to have.

Mar 14 '22 00:03 eUgEntOptIc44

I'd expect an exception handling in the loop of download.rb to prevent it from crashing when a single download fails

Mar 14 '22 11:03 eUgEntOptIc44

Thanks for reporting the error. I was offline for some days. I rerun the workflow / actions and luckily got all completed. I keep the issue open for now as a reminder to (maybe) add a retry or such to the download in the future. Cheers. Prost.

Mar 14 '22 19:03 geraldb

Prost @geraldb 🍻 Thumbs up 👍 for the fast reaction! Take care, Regards Jean-Luc

Mar 16 '22 07:03 eUgEntOptIc44

Hi @geraldb

sorry for bringing this up another time but ... it happened again. Maybe their DDOS protection kicks in at some point. Just a random idea. To be honest I got no idea. I'd be so happy if you had the time to dig into this issue.

Mar 25 '22 10:03 eUgEntOptIc44

Thanks for the notice / reminder. I kicked-off a manual (re)run and luckily this time the run / dataset update completed successfully. For sure - next time it might be time for trying to add a retry or such in the download script.

Mar 25 '22 10:03 geraldb

as noted above I tried my best to integrate exception handling myself to reduce the workload on your side.

I managed to get download.rb successfully running (after modification). However double checking for typos and flaws might be a good idea. My knowledge of ruby is close to non existing...

Mar 25 '22 14:03 eUgEntOptIc44

GitHub Actions workflow fails (HTTP 404) on downloading ek.json - Add A Retry?