mlem icon indicating copy to clipboard operation
mlem copied to clipboard

Issue listing MLEM objects on S3

Open schallerala opened this issue 2 years ago • 0 comments

Using the latest release 0.2.7, I have initialized and imported a model and as a target, used a S3 bucket (mlem init s3://bucket/awesome_project followed by mlem import ckpt/best.pkl s3://bucket/awesome_project/ckpt/best).

Then, when I try to list the models in that S3 buckets, I get the following fatal error (mlem ls s3://bucket/awesome_project):

❌ Unexpected error: _find() got an unexpected keyword argument 'recursive'
Please report it here: <https://github.com/iterative/mlem/issues>

From my understanding and digging inside mlem's code, it relies on fsspec's glob method. However, when creating a S3 location, its fs.glob method doesn't support the recursive kwargs (https://github.com/iterative/mlem/blob/HEAD/mlem/core/index.py#L86-L88), which is relayed to the fsspec's ls method (https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.glob).

Not sure why the recursive is needed when the pattern has that **.

PS. having the latest fsspec: 2022.7.1 and s3fs: 2022.7.1

Happy to help if I wasn't clear.

schallerala avatar Aug 23 '22 12:08 schallerala