kedro-plugins
kedro-plugins copied to clipboard
`kedro-datasets`: Rewrite the s3 datasets test
Description
Related: #463 #360
Context
Some xfail
tests since bumping s3fs
version -
-
tests/dask/test_parquet_dataset.py
-test_pass_credentials
-
tests/spark/test_spark_dataset.py
-test_no_version
-
tests/netcdf/test_netcdf_dataset.py
-test_pass_credentials
More tests skipped in #360 which load the dataset from s3 object -
This is because NetCDFDataset
creates a local copy of the remote data to load them in a temp_path
. This uses self._fs.get()
which is causing the tests to hang indefinitely because it doesn't play well with the mocked s3 objects.