skate icon indicating copy to clipboard operation
skate copied to clipboard

"too many open files" error

Open smarj opened this issue 2 years ago • 0 comments

I've been using skate on macOS for a few months, and recently moved to a local charm server running on Linux. I had around 400 items in a database that I exported with skate list. I've been importing them a couple of dozen at a time, and decided to import the rest today. I was getting a too many open files error. I upgraded to latest. The local charm server is now at 0.12.4 and skate is at 0.2.1. The error reporting has changed, but the result is still "too many open files".

Here is the (somewhat redacted) new error:

$ skate set ${KEY}@database.db "${VALUE}"
Error: Opening file: "/Users/smarj/Library/Application Support/charm/«LINUXSERVER»/kv/database.db/000651.sst" error: open /Users/smarj/Library/Application Support/charm/«LINUXSERVER»/kv/database.db/000651.sst: too many open files
unable to open: /Users/smarj/Library/Application Support/charm/«LINUXSERVER»/kv/database.db/000651.sst
github.com/dgraph-io/ristretto/z.OpenMmapFile
	/home/runner/go/pkg/mod/github.com/dgraph-io/[email protected]/z/file.go:80
github.com/dgraph-io/badger/v3.newLevelsController.func1
	/home/runner/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:145
runtime.goexit
	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/asm_amd64.s:1571
Usage:
  skate set KEY[@DB] VALUE [flags]

Flags:
  -h, --help   help for set

Opening file: "/Users/smarj/Library/Application Support/charm/«LINUXSERVER»/kv/database.db/000651.sst" error: open /Users/smarj/Library/Application Support/charm/«LINUXSERVER»/kv/database.db/000651.sst: too many open files
unable to open: /Users/smarj/Library/Application Support/charm/«LINUXSERVER»/kv/database.db/000651.sst
github.com/dgraph-io/ristretto/z.OpenMmapFile
	/home/runner/go/pkg/mod/github.com/dgraph-io/[email protected]/z/file.go:80
github.com/dgraph-io/badger/v3.newLevelsController.func1
	/home/runner/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:145
runtime.goexit
	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/asm_amd64.s:1571

I have 367 keys in the database now, but the open file limit was set to 256. I ran ulimit -n 2048. It's almost like skate keeps recently added keys open, because I hadn't seen this before with more keys added slowly (3-10 per day) than the open file limit.

smarj avatar Aug 28 '22 17:08 smarj