bug(fs): Listing only lists directories and not objects

[amunra]

We're using OpenDAL as an abstraction across different services.

We expect things to behave the same across S3, Azure and the filesystem, however we found that it isn't really always the case.

One instance is listing a path.

• In Azure & S3, listing returns back paths to objects.
• On the filesystem, objects aren't listed, instead only directories.

[Xuanwo]

Are you talking about op.list_with(path).recursive(true)?

[Xuanwo]

Would you like to share the opendal verison and your file tree layout and the code to list them?

[amunra]

Java code:

    @NotNull
    public static Set<String> listFiles(BlockingOperator client) {
        return client.list("/").stream()
                .map((entry) -> entry.path)
                .collect(Collectors.toSet());
    }

I'm pretty pressed between tasks at the moment and can't produce a full repro, but we dedicate a fresh new directory in our tests and read and write that directory via OpenDAL exclusively.

At certain points we list to assert side-effects and those assertions fail when using fs rather than S3 or Azure.

We've worked around this by checking for object existence by doing a read and the object is there, so I suspect it's a list issue rather than anything else.

Apologies for not being able to provide more context at this point.

Versions.

The software that writes uses:

# Cargo.toml
opendal = "0.42.0"

The software that tests uses:

<!-- pom.xml -->
<dependency>
    <groupId>org.apache.opendal</groupId>
    <artifactId>opendal-java</artifactId>
    <version>0.42.0</version>
    <scope>test</scope>
</dependency>

[Xuanwo]

Thanks a lot.

list(/) should return all entires under /. For example:

/
/file_a
/dir_b/file_c

The returning list should be ["file_a", "dir_b/"]. Are you saying that fs only returns dir_b/ but missed file_a? Will this changed after upgrading to our latest 0.44.1?

[bluestreak01]

it returns list of top-level directories for fs schema, e.g. from your example we see:

/
/file_a
/dir_b

[Xuanwo]

it returns list of top-level directories for fs schema, e.g. from your example we see:

• / should not be returned
• file_a is a file, it's ok to be listed here.
• dir_b should be present, but it's path should be dir_b/.

I will try to reproduce this and create a behavior test for this.

[Zheaoli]

The directory structure is

Permissions Size User      Date Modified Name
drwxr-xr-x     - manjusaka 11 1月  01:43 /tmp/opendal
drwxr-xr-x     - manjusaka 11 1月  01:43 ├── demo
.rw-r--r--     0 manjusaka 11 1月  01:43 │  └── abc
.rw-r--r--     0 manjusaka 11 1月  01:43 └── demo2

The java code

       System.out.println( blockingOp().list("opendal/").stream()
           .map((entry) -> entry.path)
           .collect(Collectors.toSet()));

the results is

[opendal/demo2, opendal/demo/]

[Xuanwo]

Hi @amunra, would you like to try 0.45 for verify?

[amunra]

Thanks!

It may take me a little while to get back on this one.

At the moment we're stuck with 0.42.0 since we've encountered some yet-undiagnosed bug with S3 with newer releases which only happens with live S3 instances and not our s3s-fs—based mock infra.

[Xuanwo]

It may take me a little while to get back on this one.

Got it.

At the moment we're stuck with 0.42.0 since we've encountered some yet-undiagnosed bug with S3 with newer releases which only happens with live S3 instances and not our s3s-fs—based mock infra.

Please let me know if any help we can provide.

[Xuanwo]

Hi, I'm going to close this issue now. Feel free to reopen it if the problem persists.