inotify_init1() failed: too many open files

[janisz]

trafficstars

I'm using tailer from this project. And when running it multiple times I finish with folowing error inotify_init1() failed: too many open files. I've checked the code and it looks like on Close files are not unwatched.

package tailer

import (
	"fmt"
	"github.com/fstab/grok_exporter/tailer"
	"github.com/fstab/grok_exporter/tailer/fswatcher"
	"github.com/fstab/grok_exporter/tailer/glob"
	"github.com/sirupsen/logrus"
	"io/ioutil"
	"os"
	"testing"
)

func start(path string) (fswatcher.FileTailer, error) {
	var (
		tail fswatcher.FileTailer
		err  error
	)

	g, err := glob.FromPath(path)
	if err != nil {
		return nil, fmt.Errorf("cannot create glob from path: %q", err)
	}

	tail, err = fswatcher.RunFileTailer([]glob.Glob{g}, true, false, logrus.StandardLogger())
	if err != nil {
		return nil, fmt.Errorf("cannot run rolling file tailer: %q", err)
	}

	return tailer.BufferedTailer(tail), nil
}

func TestLeakingWatch(t *testing.T) {
	for {
		f, _ := ioutil.TempFile("", "")
		f.Close()
		tf, err := start(f.Name())
		if err != nil {
			t.Fatal(err)
		}
		t.Log("OK")
		tf.Close()
		os.Remove(f.Name())
	}
}

=== RUN   TestLeakingWatch
    tailer_test.go:41: OK
  ⋮
    tailer_test.go:41: OK
time="2020-09-06T15:40:40+02:00" level=info msg="watching new file" directory=/tmp fd=3 file=196131903
    tailer_test.go:41: OK
  ⋮
    tailer_test.go:41: OK
time="2020-09-06T15:40:40+02:00" level=info msg="file was removed, closing and un-watching" directory=/tmp fd=3 file=196131903
    tailer_test.go:41: OK
    tailer_test.go:41: OK
  ⋮
    tailer_test.go:41: OK
    tailer_test.go:39: cannot run rolling file tailer: "inotify_init1() failed: too many open files"
--- FAIL: TestLeakingWatch (0.04s)
FAIL

[fstab]

The idea on Linux is that we watch only directories and not individual files, because with inotify we get all necessary events from watching the directory.

https://github.com/fstab/grok_exporter/blob/ed28cb8f23429d9f4e15faf570d31175847ea925/tailer/fswatcher/fswatcher_linux.go#L99-L102

We close the file when it's deleted, but there is nothing we need to un-watch on Linux.

Can you help me to find where all those files are opened?