GIBS-Downloader
GIBS-Downloader copied to clipboard
Add error handling for faulty downloads
For all commands that depend on the original image being downloaded properly (e.g. tiling, writing to tfrecords, generating video), implement some sort of check that the original images we are using are what we expect them to be. Things to consider:
- check that image size is as expected
- check that pixel values are as expected (not completely missing from image)
Alternatively, instead of implementing the check while tilling or writing to tfrecords AFTER the images are downloaded, catch any errors from the call to gdal_translate which may signal that the data being downloaded is faulty WHILE it is downloading.
Unexpected errors/warnings that have sometimes occurred in downloading:
-
ERROR 1: GDALWMS: Unable to download block 96, 0. URL: Failed to connect to gibs.earthdata.nasa.gov port 443: Timed out. HTTP status code: 0, error: Failed to connect to gibs.earthdata.nasa.gov port 443: Timed out.
-
Warning 1: libjpeg: Premature end of JPEG file (this warning can be turned as an error by setting GDAL_ERROR_ON_LIBJPEG_WARNING to TRUE)
I think we'd be interested in knowing what requests failed and when the occurred. That way we can see if there is a server/service issue on our end.
I think we'd be interested in knowing what requests failed and when the occurred. That way we can see if there is a server/service issue on our end.
There was a request today (3/3/21) for an image of the whole world, which should have occurred between 10am-12:30pm est which resulted in the timeout error