alibi-detect icon indicating copy to clipboard operation
alibi-detect copied to clipboard

Improve error handling for URL requests in dataset fetch methods

Open ascillitoe opened this issue 2 years ago • 3 comments

The requests.get() calls in the fetch methods such as fetch_nab and fetch_cifar10c currently don't have any error handling, potentially resulting in unclear errors when the requested url is down. For example see:

https://github.com/SeldonIO/alibi-detect/blob/436e99efb88c922dc04cec97b0b52d88bf439d65/alibi_detect/datasets.py#L369-L371

The url requests could be wrapped in try/except blocks which log an exception message and raise an error, in a similar way to the alibi datasets methods:

https://github.com/SeldonIO/alibi/blob/3a589c4de86a89fbdb956427d0d3020de05bdfda/alibi/datasets.py#L105-L110

Logic is also needed in the tests to skip tests when the URL is down, for example:

https://github.com/SeldonIO/alibi/blob/3a589c4de86a89fbdb956427d0d3020de05bdfda/alibi/tests/test_datasets.py#L13-L17

ascillitoe avatar Aug 23 '21 13:08 ascillitoe

Coming back to this. The only issue with skipping the test when a URL is down is that CI won't pick up cases where URL's are permanently dead or changed. Ideally we need to skip temporarily dead URL's during nightly and PR tests, but still check these less frequently i.e. during CI for releases.

ascillitoe avatar Sep 02 '21 11:09 ascillitoe

Fixed by #350.

jklaise avatar Sep 28 '21 13:09 jklaise

Reopening this since we missed a utility function in alibi_detect.datasets; the corruption_types_cifar10c function causes a CI failure if https://storage.googleapis.com/seldon-datasets is down (see https://github.com/SeldonIO/alibi-detect/runs/6662544833). We should add the try-except logic added in #350 to any tests using this function.

ascillitoe avatar May 31 '22 08:05 ascillitoe