kedro-plugins icon indicating copy to clipboard operation
kedro-plugins copied to clipboard

`kedro-datasets`: Rewrite the s3 datasets test

Open ankatiyar opened this issue 11 months ago • 0 comments

Description

Related: #463 #360

Context

Some xfail tests since bumping s3fs version -

  • tests/dask/test_parquet_dataset.py- test_pass_credentials
  • tests/spark/test_spark_dataset.py - test_no_version
  • tests/netcdf/test_netcdf_dataset.py- test_pass_credentials

More tests skipped in #360 which load the dataset from s3 object - This is because NetCDFDataset creates a local copy of the remote data to load them in a temp_path. This uses self._fs.get() which is causing the tests to hang indefinitely because it doesn't play well with the mocked s3 objects.

Possible Implementation

Possible Alternatives

ankatiyar avatar Feb 28 '24 16:02 ankatiyar