fsnotify
fsnotify copied to clipboard
fsnotify
short read in readEvents on Windows
I have some watcher.Errors: short read in readEvents() once every couple of minutes.
I'm not able to reproduce the problem if I run the smallest code possible to watch the same directory.
What should I try next to reproduce the short read error?
go version go1.4.2 windows/386
That error would be coming from here: https://github.com/go-fsnotify/fsnotify/blob/master/windows.go#L448
Unfortunately I don't know the Windows code very well yet. It was inherited from Chris Howey's version of fsnotify, which was inherited from winfsnotify.
If you manage to add enough to a code example to reproduce the issue, that would really help.
I have a similar problem (Windows 8.1), running golang 1.5.2 and the latest go-fsnotify (from gopkg.in/fsnotify.v1). When I place a lot of files (200+) in my directory I get the following:
2016/02/10 15:08:31 error: short read in readEvents()
2016/02/10 15:08:35 error: short read in readEvents()
2016/02/10 15:08:37 error: short read in readEvents()
2016/02/10 15:08:40 error: short read in readEvents()
2016/02/10 15:08:43 error: short read in readEvents()
2016/02/10 15:08:46 error: short read in readEvents()
2016/02/10 15:08:49 error: short read in readEvents()
2016/02/10 15:08:51 error: short read in readEvents()
2016/02/10 15:08:58 EOF
When EOF is put out, the executable crashes. Could this be something like a buffer overflow?
Sorry, I'm still not really sure. :disappointed:
Are you using just an ordinary local directory? And you say it's when there are a lot of files? With a few files it works? Maybe I can reproduce the issue. That would be a good first step.
I did some testing. The EOF was caused by a xml decoder I use, so solved that one.
However the short read error remains.
In my setup I use ftp to upload a bunch of files. The problem also occurs when I just copy the files in the folder. When a "WRITE" event is fired (it fires three times btw, don't know if this is normal) I decode the content of each file and put that content in a db. So quite heavy work. Could this be interfering with fsnotify's ability to process all the new files? Or perhaps because it's a lot to process the os takes a different route which throws off fsnotify or the underlying API? Just speculating btw :)
Ok, had the opportunity to test on a Linux machine and had a weird difference:
On Windows the WRITE event is fired 3x, in Linux (ElementaryOS) 1x
When a short read error occurs (on Windows) it seems to truncate the file it is reading (not necessarily always the case, seems to happen sometimes). This is also why my XML decoder errored with EOF.
I just use a normal directory. To trigger the WRITE event I just copy the files to the watched folder. Hope it helps, if you need any other info please let me know.
When I use FTP (on Windows) with a slow connection speed and I read the file on the first WRITE event, the file is not complete. This makes the XML decoder fail.
Thanks for the info. I'm not sure when I (or someone else) will have a chance to dig into the Windows code for fsnotify to figure out what's going on. But any details to help reproduce it are helpful.
we were able to fix this by making the windows i/o buffer dynamically adjustable, as in https://github.com/tilt-dev/fsnotify/pull/5
This bug is not fully fixed. I'm facing the exact behavior with Windows 10 while using Go 1.17.2. Any news so far?
I encountered the same issue with GO 1.16. Windows Server 2012 R2. According to tilt-dev#5, if I am understanding it correctly, does it mean that we can set the buffer size however we need? And yet I cannot find the code changes from that commit in the current main/master branch
we were able to fix this by making the windows i/o buffer dynamically adjustable, as in tilt-dev#5
I believe the reason why this is not fixed yet is because the fix above was not merged into the upstream library? Is there any chance of the fix getting merged upstream (in fsnotify/fsnotify, and not just in the tilt-dev/fsnotify fork)?
@gnoack I think this issue sums up the current merge queue / mainenance situation - https://github.com/fsnotify/fsnotify/issues/413
@gnoack I think this issue sums up the current merge queue / mainenance situation - https://github.com/fsnotify/fsnotify/issues/413
It's now maintained again @nicks!
I looked at your patch in https://github.com/tilt-dev/fsnotify/pull/5, but it doesn't really "dynamically" adjust the buffer size: people will have to call SetBufferSize(), which is fine for a "quick fix" but not ideal, especially since it'll only be available on Windows (e.g. in tilt you need to use build tags to actually call it).
It would be better to increase the buffer size as needed; the windows code is a bit messy, but I'm sure this has got to be possible.
Do you want to work on a patch for this?
Also: I wonder if we can't just double the buffer size to 8K or 16K or something, which should at least make things a bit better. It looks like tilt uses 64K by default.
