packr

packr copied to clipboard

Hi!

At Gitea we'd like to move from go-bindata to packr2. However, we're noticing that on attempting to run packr we're requiring thousands of open fds.

I've looked for obvious leaks but if I had to guess I suspect that something isn't closing the Body of a http.Response

https://github.com/go-gitea/gitea/pull/5952 is the Gitea pr adding packr2.

A PR would be very welcome.

---

Mark Bates On Feb 8, 2019, 2:42 AM -0500, zeripath [email protected], wrote:

go-gitea/gitea#5952 is the Gitea pr — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

If I knew where the issue was I'd happily write one. I suspect it's somewhere in one of the dependent packages but I was hoping you might have an idea where.

I haven’t run across the problem in any of my apps or usage, so I don’t even know how to reproduce. If you could provide a simple app that reproduces the issue, that would help.

---

Mark Bates On Feb 8, 2019, 7:35 AM -0500, zeripath [email protected], wrote:

If I knew where the issue was I'd happily write one. I suspect it's somewhere in one of the dependent packages but I was hoping you might have an idea where. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

Also, if you could provide more details on things like what OS and/or browser? Does it happen when packing files? Does it happen during web requests? How are you packing (any flags or custom stores)? How does the problem manifest, etc...

---

Mark Bates On Feb 8, 2019, 7:35 AM -0500, zeripath [email protected], wrote:

If I knew where the issue was I'd happily write one. I suspect it's somewhere in one of the dependent packages but I was hoping you might have an idea where. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

So you need to set ulimit -n appropriately low and run packr2 to generate packed files on an appropriately large package. I'm running Ubuntu Linux with the default ulimit of 1024.

In terms of minimal test cases, it's fairly difficult as I don't know exactly what packr2 is doing. The gitea pr attached shows an example where packr has open >1024 fds. Basically in order to show it you do make generate and it will call packr2.

Looking at the pr I don't think we're adding anything special to packr2. We tried just running packr2 from the root of the Gitea checkout directory but that required even more file descriptors.

Yes it's running over a lot of files but I can't see how packr2 or why it would keep these files open. Is it possible that one of the upstream packages of packr2 isn't closing the Body of a http.Response? That seems to be the canonical way of getting go to leak to file descriptors.

Not sure what http.Response would have to do with packing files. I’m currently dealing with a death in the family, so I’ll try and look at it on monday.

---

Mark Bates On Feb 8, 2019, 8:56 AM -0500, zeripath [email protected], wrote:

Looking at the pr I don't think we're adding anything special to packr2. We tried just doing packr2 from the root directory but that required even more file descriptors. Yes it's running over a lot of files but I can't see how packr2 or why it would keep these files open. Is it possible that one of the upstream packages of packr2 isn't closing the Body of a http.Response? That seems to be the canonical way of getting go to leak to file descriptors. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

Oh I'm sorry to hear that. I hope you're doing as well as can be expected.

No rush.

If I get a chance I'll try to work it out and send a PR if I can.

I've had a chance to play with this, and I can't reproduce on a Mac or docker, I don't have a linux box. On my Mac, the lowest I could go with ulimit was 300, otherwise my system wouldn't function. :) In Docker I could get down to 200, any lower and go get fails.

I ran all of this against github.com/gobuffalo/buffalo which has a lot of dependencies to search for boxes. It ends up packing 17 boxes in total.

I also tried both of those environments against the PR you sent and also couldn't reproduce.

Without taking random stabs in the dark, I'm not sure there's much I can do since I can't reproduce on my end.

If you can reproduce, then you're in the best position to help fix it. The https://github.com/gobuffalo/packd might be a good place to look as well, since it does a lot of file stuff there.

Hmm... Thanks for looking at this!

I upgraded to go 1.11 and the problem disappears. I was on go 1.10 I wonder if that's what the problem was?

Thanks for your help.

I think I'll close the issue.

Not sure. I only tested on 1.10. Glad the issue went away. :)

I spoke too soon. It came back as soon as I did a clean and remake of packr2.

It's weird you can't reproduce. To get packr2 to do anything on the pr I mentioned, you have to run: TAGS="bindata" make clean generate build

I just tested it, I think I can reproduce on my Linux:

$ go version
go version go1.11.5 linux/amd64
$ ulimit -n
1024
$ env TAGS="bindata" make clean generate build
packr2 clean
go clean -i ./...
rm -rf gitea dist modules/{options,public,templates}/*-packr.go modules/{options,public,templates}/packrd/* \
	integrations*.test \
	integrations/gitea-integration-pgsql/ integrations/gitea-integration-mysql/ integrations/gitea-integration-sqlite/ integrations/gitea-integration-mssql/ \
	integrations/indexers-mysql/ integrations/indexers-pgsql integrations/indexers-sqlite integrations/indexers-mssql \
	integrations/mysql.ini integrations/pgsql.ini integrations/mssql.ini
can't load package: package github.com/go-gitea/gitea: code in directory /home/michalakst/go/src/github.com/go-gitea/gitea expects import "code.gitea.io/gitea"
go generate github.com/go-gitea/gitea/cmd github.com/go-gitea/gitea/integrations github.com/go-gitea/gitea/integrations/migration-test github.com/go-gitea/gitea/models github.com/go-gitea/gitea/models/migrations github.com/go-gitea/gitea/modules/auth github.com/go-gitea/gitea/modules/auth/ldap github.com/go-gitea/gitea/modules/auth/oauth2 github.com/go-gitea/gitea/modules/auth/openid github.com/go-gitea/gitea/modules/auth/pam github.com/go-gitea/gitea/modules/avatar github.com/go-gitea/gitea/modules/base github.com/go-gitea/gitea/modules/cache github.com/go-gitea/gitea/modules/context github.com/go-gitea/gitea/modules/cron github.com/go-gitea/gitea/modules/generate github.com/go-gitea/gitea/modules/gzip github.com/go-gitea/gitea/modules/highlight github.com/go-gitea/gitea/modules/httplib github.com/go-gitea/gitea/modules/indexer github.com/go-gitea/gitea/modules/lfs github.com/go-gitea/gitea/modules/log github.com/go-gitea/gitea/modules/mailer github.com/go-gitea/gitea/modules/markup github.com/go-gitea/gitea/modules/markup/csv github.com/go-gitea/gitea/modules/markup/external github.com/go-gitea/gitea/modules/markup/markdown github.com/go-gitea/gitea/modules/markup/orgmode github.com/go-gitea/gitea/modules/metrics github.com/go-gitea/gitea/modules/minwinsvc github.com/go-gitea/gitea/modules/notification github.com/go-gitea/gitea/modules/notification/base github.com/go-gitea/gitea/modules/notification/indexer github.com/go-gitea/gitea/modules/notification/mail github.com/go-gitea/gitea/modules/notification/ui github.com/go-gitea/gitea/modules/options github.com/go-gitea/gitea/modules/pprof github.com/go-gitea/gitea/modules/private github.com/go-gitea/gitea/modules/process github.com/go-gitea/gitea/modules/public github.com/go-gitea/gitea/modules/recaptcha github.com/go-gitea/gitea/modules/search github.com/go-gitea/gitea/modules/setting github.com/go-gitea/gitea/modules/ssh github.com/go-gitea/gitea/modules/sync github.com/go-gitea/gitea/modules/templates github.com/go-gitea/gitea/modules/test github.com/go-gitea/gitea/modules/user github.com/go-gitea/gitea/modules/util github.com/go-gitea/gitea/modules/validation github.com/go-gitea/gitea/routers github.com/go-gitea/gitea/routers/admin github.com/go-gitea/gitea/routers/api/v1 github.com/go-gitea/gitea/routers/api/v1/admin github.com/go-gitea/gitea/routers/api/v1/convert github.com/go-gitea/gitea/routers/api/v1/misc github.com/go-gitea/gitea/routers/api/v1/org github.com/go-gitea/gitea/routers/api/v1/repo github.com/go-gitea/gitea/routers/api/v1/swagger github.com/go-gitea/gitea/routers/api/v1/user github.com/go-gitea/gitea/routers/api/v1/utils github.com/go-gitea/gitea/routers/dev github.com/go-gitea/gitea/routers/org github.com/go-gitea/gitea/routers/private github.com/go-gitea/gitea/routers/repo github.com/go-gitea/gitea/routers/routes github.com/go-gitea/gitea/routers/user github.com/go-gitea/gitea/routers/user/setting github.com/go-gitea/gitea/routers/utils
Error: open /home/michalakst/go/src/github.com/go-gitea/gitea/public/vendor/plugins/emojify/images/saxophone.png: too many open files
Usage:
  packr2 [flags]
  packr2 [command]

Available Commands:
  build       Wraps the go build command with packr
  clean       removes any *-packr.go files
  fix         will attempt to fix a application's API to match packr version v2.0.1
  help        Help about any command
  install     Wraps the go install command with packr
  version     shows packr version

Flags:
  -h, --help               help for packr2
      --ignore-imports     when set to true packr won't resolve imports for boxes
      --legacy             uses the legacy resolution and packing system (assumes first arg || pwd for input path)
      --silent             silences all output
      --store-cmd string   sub command to use for packing
  -v, --verbose            enables verbose logging

Use "packr2 [command] --help" for more information about a command.

modules/public/public.go:7: running "packr2": exit status 1
make: *** [Makefile:96: generate] Error 1

Again, I can’t reproduce. It looks like you have an error right at the beginning with a bad import. Maybe that’s the problem??

---

Mark Bates On Feb 10, 2019, 4:37 PM -0500, Stanislas Michalak [email protected], wrote:

Reopened #158. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

@stanislas-m if you move src/github.com/go-gitea/gitea to src/code.gitea.io/gitea you should be able to avoid the bad import.

I'm running go version go1.11.5 darwin/amd64 and am able to get this to trigger when I set ulimit -n to 1024.

I still can reproduce after moving the provided sources to the correct dir. I can't reproduce every time though, it seems you have to keep your system busy enough to be able to produce the error.

$ env TAGS="bindata" make clean generate build
packr2 clean
go clean -i ./...
rm -rf gitea dist modules/{options,public,templates}/*-packr.go modules/{options,public,templates}/packrd/* \
	integrations*.test \
	integrations/gitea-integration-pgsql/ integrations/gitea-integration-mysql/ integrations/gitea-integration-sqlite/ integrations/gitea-integration-mssql/ \
	integrations/indexers-mysql/ integrations/indexers-pgsql integrations/indexers-sqlite integrations/indexers-mssql \
	integrations/mysql.ini integrations/pgsql.ini integrations/mssql.ini
go generate code.gitea.io/gitea code.gitea.io/gitea/cmd code.gitea.io/gitea/models code.gitea.io/gitea/models/migrations code.gitea.io/gitea/modules/auth code.gitea.io/gitea/modules/auth/ldap code.gitea.io/gitea/modules/auth/oauth2 code.gitea.io/gitea/modules/auth/openid code.gitea.io/gitea/modules/auth/pam code.gitea.io/gitea/modules/avatar code.gitea.io/gitea/modules/base code.gitea.io/gitea/modules/cache code.gitea.io/gitea/modules/context code.gitea.io/gitea/modules/cron code.gitea.io/gitea/modules/generate code.gitea.io/gitea/modules/gzip code.gitea.io/gitea/modules/highlight code.gitea.io/gitea/modules/httplib code.gitea.io/gitea/modules/indexer code.gitea.io/gitea/modules/lfs code.gitea.io/gitea/modules/log code.gitea.io/gitea/modules/mailer code.gitea.io/gitea/modules/markup code.gitea.io/gitea/modules/markup/csv code.gitea.io/gitea/modules/markup/external code.gitea.io/gitea/modules/markup/markdown code.gitea.io/gitea/modules/markup/orgmode code.gitea.io/gitea/modules/metrics code.gitea.io/gitea/modules/minwinsvc code.gitea.io/gitea/modules/notification code.gitea.io/gitea/modules/notification/base code.gitea.io/gitea/modules/notification/indexer code.gitea.io/gitea/modules/notification/mail code.gitea.io/gitea/modules/notification/ui code.gitea.io/gitea/modules/options code.gitea.io/gitea/modules/pprof code.gitea.io/gitea/modules/private code.gitea.io/gitea/modules/process code.gitea.io/gitea/modules/public code.gitea.io/gitea/modules/recaptcha code.gitea.io/gitea/modules/search code.gitea.io/gitea/modules/setting code.gitea.io/gitea/modules/ssh code.gitea.io/gitea/modules/sync code.gitea.io/gitea/modules/templates code.gitea.io/gitea/modules/test code.gitea.io/gitea/modules/user code.gitea.io/gitea/modules/util code.gitea.io/gitea/modules/validation code.gitea.io/gitea/routers code.gitea.io/gitea/routers/admin code.gitea.io/gitea/routers/api/v1 code.gitea.io/gitea/routers/api/v1/admin code.gitea.io/gitea/routers/api/v1/convert code.gitea.io/gitea/routers/api/v1/misc code.gitea.io/gitea/routers/api/v1/org code.gitea.io/gitea/routers/api/v1/repo code.gitea.io/gitea/routers/api/v1/swagger code.gitea.io/gitea/routers/api/v1/user code.gitea.io/gitea/routers/api/v1/utils code.gitea.io/gitea/routers/dev code.gitea.io/gitea/routers/org code.gitea.io/gitea/routers/private code.gitea.io/gitea/routers/repo code.gitea.io/gitea/routers/routes code.gitea.io/gitea/routers/user code.gitea.io/gitea/routers/user/setting code.gitea.io/gitea/routers/utils
Error: open /home/michalakst/go/src/code.gitea.io/gitea/public/vendor/plugins/emojify/images/large_orange_diamond.png: too many open files
Usage:
  packr2 [flags]
  packr2 [command]

Available Commands:
  build       Wraps the go build command with packr
  clean       removes any *-packr.go files
  fix         will attempt to fix a application's API to match packr version v2.0.1
  help        Help about any command
  install     Wraps the go install command with packr
  version     shows packr version

Flags:
  -h, --help               help for packr2
      --ignore-imports     when set to true packr won't resolve imports for boxes
      --legacy             uses the legacy resolution and packing system (assumes first arg || pwd for input path)
      --silent             silences all output
      --store-cmd string   sub command to use for packing
  -v, --verbose            enables verbose logging

Use "packr2 [command] --help" for more information about a command.

modules/public/public.go:7: running "packr2": exit status 1
make: *** [Makefile:96: generate] Error 1

I had the same problem on my Mac (macOS 10.14.6, go1.13 and buffalo v0.14.10), but only with the command buffalo build, with buffalo dev it worked. Since building on the server (Github Actions, same go and buffalo versions) worked, I just increased the limit of number of open files like discribed here: https://medium.com/mindful-technology/too-many-open-files-limit-ulimit-on-mac-os-x-add0f1bfddde That solved the problem for me.