prefix option of S3FileSystem.find() doesn't work

[nkwook]

Hello, im using s3fs in python 3.11

prefix parameter of https://s3fs.readthedocs.io/en/latest/api.html#s3fs.core.S3FileSystem.find seems not working.

Even the API needs to return filtered file name list, it just returns all filenames on the folder.

Thank you in advance.

[martindurant]

Please post an example of the call you are making, what you expect and what you get instead. If this is in the form of a test function, even better!

[nkwook]

    def get_file_lists(self, prefix):
        self.logger.info(prefix)
        result = []
        try:
            result = self.s3f.find(
                f'{os.environ["S3_BUCKET_NAME"]}', prefix=prefix
            )

        except Exception as e:
            self.logger.error(e)
        return result

I was using this function for filter files with prefix, but however when I call this now, find function returns entire filenames in os.environ["S3_BUCKET_NAME"]. So I need to add following code to achieve my goal.

      # temp code since prefix param not working
      # result = [x for x in result if x.startswith(f'{os.environ["S3_BUCKET_NAME"]}/'+prefix)]

Thank you!

[martindurant]

@rlamy ; probably a consequence of the directory iterator that the prefix got lost? Perhaps you have a little time to have a look.

[rlamy]

@martindurant I couldn't reproduce the issue. I also don't really see how the _iterdir() change could affect this.

[martindurant]

I suggested because iterdict was the last to touch the code affecting this functionality.

@nkwook , do you think you can format your code in the style of one of our tests (i.e., no environment variasbles and having explicitly set up the files you expect to search though) - it should fail against the mock moto server just the same as against AWS. I am also now trying it and am NOT seeing a problem.

In [10]: fs.find("mymdtemp/zarr", prefix="o")
Out[10]: ['mymdtemp/zarr/oi', 'mymdtemp/zarr/oi.ipynb']

In [11]: len(fs.find("mymdtemp/zarr"))
Out[11]: 1291