Very slow startup due to directory crawling at import time

[jtackaberry]

I noticed my application taking a fairly long time to start up and the cause seems to be packr aggressively crawling the hierarchy relative to the current directory at import time. On a project with a sizeable node_modules directory served over NFS, this quickly adds up.

It can be reproduced with this trivial program:

package main

import (
        _ "github.com/gobuffalo/packr/v2"
)

func main() {
}

Executing that from my project directory containing node_modules, we have several thousand newfstatat() syscalls:

$ strace ~/t 2>&1 | grep newfstatat | wc -l
2822

Is this expected?

OS is Ubuntu 18.04.1 with kernel 4.18.0-15-generic.

[timraymond]

Hi @jtackaberry ,

I attempted to reproduce this issue in Docker since I'm on a Mac. Interestingly, I don't get the same output (using the ubuntu 18.04 image):

$ docker run --rm --cap-add SYS_PTRACE packr:issue-196
16

The code, Dockerfile, etc. are available here: https://github.com/timraymond/packr-issue-196 and the image I'm running is here: https://cloud.docker.com/repository/docker/timraymond/packr-issue-196 . Mind taking a look and making sure that what I'm doing aligns with what you're doing? Please feel free to open PRs there if you need to make some changes to accurately reproduce the issue.

[invidian]

It seems that this issue only occurs, when packr runs from source directory. If you use different PWD, issue does not occur. I guess this is development feature described in https://github.com/gobuffalo/packr/tree/master/v2#development-made-easy.

In any case, I created a PR to https://github.com/timraymond/packr-issue-196, which exposes the issue. See https://github.com/timraymond/packr-issue-196/pull/1.

I think packr during importing should only traverse registered boxes rather than all files, which might be a lot.

As a workaround, just switch to directory different from building directory and then it should be fine.

[timraymond]

I've merged @invidian 's PR to my repro app FYI. That should be a good place to start investigating this.

[ChinaHDJ1]

remove code _ "github.com/gobuffalo/packr/v2"