SPIFFS_remove() doesn't free file descriptor on success

[jcwren]

I have a file system I created a handful of files in. I have 4 file descriptors available. On the 5th call to SPIFFS_remove(), I get SPIFFS_ERR_OUT_OF_FILE_DESCS (-10007). Looking at the code, it appears that SPIFFS_remove() only calls spiffs_fd_return() when there's an error, but never on success.

[pellepl]

Hmm, that is weird. I ran this in the testbench without problems. There are less than 32 filedescs in the setup. What's you config?

 TEST(remove_eats_fd_152) {
  int res, i, j;
  fs_reset_specific(0, 0, 128*1024, 4096, 4096, 128);
  for (j = 0; j < 32; j++) {
    char fname[32];
    sprintf(fname, "file%i.txt", j);
    spiffs_file fd = SPIFFS_open(FS, fname, SPIFFS_O_RDWR | SPIFFS_O_CREAT, 0);
    TEST_CHECK_GT(fd, 0);
    for (i = 0; i < 32; i++) {
      char *buf = "I'm gonna eat your descriptor if you ever remove me, haha\n";
      res = SPIFFS_write(FS, fd, buf, strlen(buf));
      TEST_CHECK_GE(res, SPIFFS_OK);
    }
    res = SPIFFS_close(FS, fd);
    TEST_CHECK_EQ(res, SPIFFS_OK);
  }
  for (j = 0; j < 32; j++) {
    char fname[32];
    sprintf(fname, "file%i.txt", j);
    res = SPIFFS_remove(FS, fname);
    TEST_CHECK_EQ(res, SPIFFS_OK);
  }
  return TEST_RES_OK;
} TEST_END

[jcwren]

Here's the config: spiffs_config.txt

(Amazing that github doesn't let you attach a .h file...)

Looking at SPIFFS_remove(), how does the file handle it allocates ever get released?

[pellepl]

I agree, this does not look symmetrical at all. Probably I either made a 1) uncommented optimisation, or 2) a bug. ~~Contrary to user callls, the actual fd struct is not collected from the descriptor space given from user, but is kept on stack.~~ (sorry, being delirious here, ignore that) I'll look into it. In the meantime, in your use case: is it fixed by calling spiffs_fd_return regardless result?

[pellepl]

Ahh, right! No, spiffs_fd_return is not needed when result is OK. On successful truncate/remove, spiffs_object_truncate will trigger a spiffs_cb_object_event. This function will recognize the full removal and thus free the fd internally.

So the bug must be something else.

[jcwren]

I will check this tomorrow. It seems I remember some other functions are calling spiffs_fd_return on an OK, though, which is I why I thought this one was wonky. I'll see if I can find that one tomorrow also.

[barbiomalefico]

Hi, I got the same problem. I was able to delete 2 files and after that I always receive SPIFFS_ERR_OUT_OF_FILE_DESCS (-10007). A new mount grant me the capability to delete other 2 files. Looking at the function spiffs_fd_find_new I noticed that it uses the fs->fd_count wich value was 2. Then looking at the SPIFFS_mount I saw that fs->fd_count = (fd_space_size/sizeof(spiffs_fd));. So i have increased the size of fd_space_size of 100 times and now I'm able to delete up to 100 files. I thought the fd_space_size indicated the number of file descriptors that I can open together, not the maximum number of file I can have on the filesystem. Probably I was wrong about it, but this limitation didn't take place when I cretate file (I was able to create 1000 files). I hope this can help to solve the problem. My workaround now is to remount the fs after 100 deletes.

[pellepl]

I'll look into it. Fd_space_size is the fd buffer in bytes given by user IIRC. A delete shall not consume fd:s.

Den 3 aug. 2017 11:17 skrev "barbiomalefico" [email protected]:

Hi, I got the same problem. I was able to delete 2 files and after that I always receive SPIFFS_ERR_OUT_OF_FILE_DESCS (-10007). A new mount grant me the capability to delete other 2 files. Looking at the function spiffs_fd_find_new I noticed that it uses the fs->fd_count wich value was 2. Then looking at the SPIFFS_mount I saw that fs->fd_count = (fd_space_size/sizeof(spiffs_fd));. So i have increased the size of fd_space_size of 100 times and now I'm able to delete up to 100 files. I thought the fd_space_size indicated the number of file descriptors that I can open together, not the maximum number of file I can have on the filesystem. Probably I was wrong about it, but this limitation didn't take place when I cretate file (I was able to create 1000 files). I hope this can help to solve the problem. My workaround now is to remount the fs after 100 deletes.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/pellepl/spiffs/issues/152#issuecomment-319915345, or mute the thread https://github.com/notifications/unsubscribe-auth/AE3NVzjGMOZHLebYIsrUsUQ3aywBGBIMks5sUZBGgaJpZM4N49ql .

[vincent-d]

I have the same issue, fixed by calling spiffs_fd_return whatever the result of spiffs_object_truncate is.

[pellepl]

Are you guys calling SPIFFS_fclose after SPIFFS_fremove?

[vincent-d]

~~Crap, I don't.~~

~~Sorry.~~

Actually, I use SPIFFS_remove not SPIFFS_fremove so I can't close

[barbiomalefico]

the topic is about SPIFFS_remove not SPIFFS_fremove and I use SPIFFS_remove

[pellepl]

Ah sorry - I'm obviously getting old. Will have a peek at it this evening.

What are your configs? When running in the testbench I can at least remove 1536 files on 8 fds using SPIFFS_remove without problem.

[pellepl]

Hmm.. I am now running a config exactly as @jcwren - still, I cannot reproduce it. I can remove numerous files using SPIFFS_remove without hitting any fd cap... Obviously, I need more info as I am missing something here. I'll see if I can put together some debug functionality for printing needed stuff, if any of you could run that on your target.

What hardware and compiler are you seeing this on?

[pellepl]

Hmm... Are all of you running with SPIFFS_TEMPORAL_FD_CACHE = 1? If so, can anyone try disabling SPIFFS_TEMPORAL_FD_CACHE and see if that changes anything?

[barbiomalefico]

in my configuration I have SPIFFS_TEMPORAL_FD_CACHE = 1 and, after putting it to 0, I don't see the problem anymore.

[vincent-d]

On my config, SPIFFS_TEMPORAL_FD_CACHE is disabled

[pellepl]

@barbiomalefico ok good, then I think I found the problem. @vincent-d errr... does not compute :/ I'll update stuff in ca 8 hours (after work). With that patch, things should work with SPIFFS_TEMPORAL_FD_CACHE enabled. @vincent-d do you have a small example how to reproduce your problem?

[pellepl]

Ppl, could you please try with this latest patch?

[valkuc]

I have faced same issue recently. After update to current master head the issue not reproduce anymore.

[MightyPork]

Facing the same crash right now in 0.3.7, here is a testcase which reproduces it: https://share.ondrovo.com/2020-07-09/fs_sim.zip

cmake . && make && ./fsim

It takes a long time, about 10 minutes. It crashes when the counter reaches about 21'000. Disabling SPIFFS_TEMPORAL_FD_CACHE seems to fix the problem.