Panic on Linux: No space left on device

[uazu]

None of my drives are over 50% in use, so the error makes no sense. I just ran RUST_BACKTRACE=1 timetrack track with no change to the default config. Rust is up to date. Here's the output:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Io(Os { code: 28, kind: Other, message: "No space left on
device" })', libcore/result.rs:945:5
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::print
             at libstd/sys_common/backtrace.rs:71
             at libstd/sys_common/backtrace.rs:59
   2: std::panicking::default_hook::{{closure}}
             at libstd/panicking.rs:211
   3: std::panicking::default_hook
             at libstd/panicking.rs:227
   4: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:511
   5: std::panicking::continue_panic_fmt
             at libstd/panicking.rs:426
   6: rust_begin_unwind
             at libstd/panicking.rs:337
   7: core::panicking::panic_fmt
             at libcore/panicking.rs:92
   8: core::result::unwrap_failed
   9: timetrack::track::<impl timetrack::TimeTracker<'a>>::track
  10: timetrack::main
  11: std::rt::lang_start::{{closure}}
  12: std::panicking::try::do_call
             at libstd/rt.rs:59
             at libstd/panicking.rs:310
  13: __rust_maybe_catch_panic
             at libpanic_unwind/lib.rs:105
  14: std::rt::lang_start_internal
             at libstd/panicking.rs:289
             at libstd/panic.rs:392
             at libstd/rt.rs:58
  15: main
  16: __libc_start_main
  17: _start

[JoshMcguigan]

Can you run timetrack config and post what it displays? Also, what operating system are you on?

[uazu]

Linux -- it says in the title. The config output is below. I wonder whether it is a notify crate bug. I've tried inotifywait -m -r /home/jim (from Debian package inotify-tools), and that runs without problem. Could it perhaps be some symlinks (or something) confusing the notify crate?

TimeTrack Configuration
    User configuration: "/home/jim/.config/timetrack/timetrack_config"
    Tracking paths: ["/home/jim"]
    Raw data: "/home/jim/.local/share/timetrack/.timetrack_raw"
    Processed data: "/home/jim/.local/share/timetrack/.timetrack_processed"

[uazu]

Just some additional notes. It appears that inotify requires that a watcher is placed on each directory (it can't watch a whole tree in one go). There are 5800+ directories under my home dir. There are also 24 symlinks to directories. So if notify is hitting any limits or bugs due to these counts, that could explain it. (But the inotifywait tool in Debian can handle it fine.)

[uazu]

If I do find . -type d it works fine. If I do find -L . -type d it reports "File system loop detected" and goes on forever. The biggest one there is .wine/dosdevices/z: linking to /. Really notify crate should be able to handle symlinks to ancestor directories as they have their uses. I tried stracing timetrack track and it stops after installing about 8000 inotify watchers. So it must have been following the symlinks (and therefore was trying to watch the entire filesystem). I guess not following the symlinks would be a safer approach.

[JoshMcguigan]

Thanks for digging into this. PR #4 will ensure TimeTrack doesn't crash when adding the watcher fails I think, but there are other issues within TimeTrack when watching the entire file system. TimeTrack expects each file change event to be for a file which is within the track path (not necessarily a direct child).

Do you have any suggestions on how TimeTrack could handle this better? Otherwise it sounds like this issue should be reported to the notify crate.

[uazu]

I don't want it to watch the entire filesystem. I just want it to watch /home/jim (in this instance). So if you can specify to notify that symlinks shouldn't be followed, then that would solve the problem. (If notify allows that.)

Another approach might be to have a config specification to 'prune' some directories out of the tree that contain subdirs that are problematic (e.g. .wine in this case), or that aren't worth following (e.g. .git folders).

Another thing to note is that on Linux, the user might need to increase /proc/sys/fs/inotify/max_user_watches if they have a large tree to watch. I think that was the limit that it was hitting, due to it following the symlink to /.

If I expand that to 20,000 then now I get a panic with "Permission Denied", because it's following the z: symlink to / and trying to watch directories it doesn't have access to. I think notify needs fixing with an option to skip directories that the user has no access to.

So that's three possible issues for notify (if not already supported): an option to not follow symlinks, an option to skip directories where the user has no access, and an option to prune matches.

You might want to keep an issue open for these Linux issues until you get the necessary support from notify, because I'm sure other people will come across the same problems.

[xfbs]

Ah, I ran into the same issue.

I patched src/config/mod.rs with

println!(                                                                                                                                                                    
    "Error {} adding watcher for path {:?}: {:?}",                                                                                                                           
    err,                                                                                                                                                                     
    track_path.to_string_lossy(),                                                                                                                                            
    err                                                                                                                                                                      
);

To show me the full error, and running cargo run -- config gives me:

Error other os error adding watcher for path "/home/xfbs": Io(Os { code: 28, kind: Other, message: "No space left on device" })

How can we fix it?

[JoshMcguigan]

@xfbs if your issue has the same cause (symlinks under your home directory to / or similar) then I think there are a couple options

0. Remove these symlinks from your system
1. Configure timetrack to watch a directory which doesn't contain these symlinks
2. Patch timetrack to allow configuration to skip certain directories, and use this to ignore the symlinks (I'd accept a PR for this)