Enumerating directory and opening files

[rocuh]

In another use case, we want to open every file in the directory (to read some info) and print this out. There's quite a lot of files (100s) and this can take a long time - due to the fact that lfs_file_open will cause another directory traverse to find the top block of the file.

Again its adding to complexity but it might be useful to have an open that you could pass an lfs_info structure that is returned from lfs_dir_read and have it open it without the search overhead (e.g. lfs_file_open_direct).

The lfs_info struct would have to be modified to store the block info I guess.

[geky]

I was thinking about what this would look like.

Instead of adding another set of filesystem functions, what if we added a magic string to the lfs_info struct so it could be passed as a filename?

The info struct would look something like this:

struct lfs_info {
    char ref[4];                 // "\x1bref" magic string
    lfs_block_t dir[2];          // directory pair
    lfs_off_t off;               // offset of entry in directory
    
    uint8_t type;                // type (reg vs dir)
    lfs_size_t size;             // size (of reg files)
    char name[LFS_NAME_MAX+1];   // name of file
};

And you could pass it directly to lfs_file_open or lfs_remove or anything that takes a path:

lfs_dir_open(&lfs, &dir, "/");
while (lfs_dir_read(&lfs, &dir, &info) == 1) {
    if (info.type == LFS_TYPE_REG) {
        lfs_file_open(&lfs, &file, (char*)&info, LFS_O_WRONLY | LFS_O_TRUNC);
        // blablabla
    }
}
lfs_dir_close(&lfs, &dir);

Thoughts?

One big concern is that references to directories can very quickly become invalidated if a directory goes bad in a different function. So using this in a multi-threaded environment could be a nightmare. Fortunately these are embedded systems, so the filesystem operations are likely to be very controlled.

[rocuh]

Yes that would great.

[apmorton]

using tricks like this makes it very difficult for the compiler to optimize away unused code.

if you add a separate function (lfs_file_open_direct for example), and I don't use it in my application, the linker can simply drop it on the floor.

if you add a conditional inside lfs_file_open to check for some magic filename and then separately handle that case, that code will be very difficult (if not impossible) for the compiler to remove, since it most likely will be unable to prove the branch is never taken.

[mon]

Just bumping this with some specific pain-points.

To iterate a directory and open files, I need to snprintf into a temporary buffer of length LFS_NAME_MAX to join the directory itself and the filename.

I know that any valid file can't have longer than LFS_NAME_MAX characters, but because the lfs_info struct has the filename as LFS_NAME_MAX characters too, GCC will warn about possible overflow into the temporary path buffer.

So I can either try and trick GCC with some casts or make the temporary buffer bigger than required - it's a bit of a mess. Having the proposed lfs_file_open_direct function would be tremendous for usability.

[geky]

Noted. I'm sold on adding a set of lfs_file_open_direct ish functions.

It looks like the standard POSIX naming for these sort of functions is openat [1]. So my vote would be:

• int lfs_file_openat(lfs_t *lfs, lfs_file_t *file, const lfs_dir_t *dir, const char *relpath, uint32_t flags)
• int lfs_dir_openat(lfs_t *lfs, lfs_dir_t *dir, const lfs_dir_t *dir, const char *relpath)
• int lfs_stat(lfs_t *lfs, const lfs_dir_t *dir, const char *relpath, struct lfs_info *info)
• etc

I've been looking at WebAssembly recently and it's interesting to note that in reinventing POSIX, the standard WASI API provides openat, and only openat (though their primary goal isn't user-friendliness): https://github.com/bytecodealliance/wasmtime/blob/master/docs/WASI-api.md#__wasi_path_open

---

The next question is when will this be added, and I'm not sure. I think it's a good idea, but there's also pressing issues around scalability and performance that need to be fixed.

I don't think I will be able to work on this any-time soon, but I will keep it on the roadmap for the future.

[geky]

Note the most complex part will be getting the current path-walk logic to behave with relative paths.

[zrno]

any update on this? Just wanted to check before I start implementing this feature in our littlefs wrapper.