Parallel walk?

[pcj]

It seems like gazelle should be theoretically parallelizable when walking the filesystem, as separate subtrees are supposed to be independent. However, it would require that things like config access and other operations are protected against concurrent writes, etc.

I assume this has been thought of, any particular design decisions on this? cc @jayconrod

[robfig]

I can't think of any reason that this wouldn't work, other than being some significant implementation work.

[alexeagle]

We're discussing this FR right now in the gazelle contributors meeting.

We'd like to add a profiler first so it's more clear to us/users whether the serial tree-walk is the bottleneck. @robfig do you have any input that could help someone contribute this? Some rough design sketch or pointers where you would make the change?

[achew22]

Go has a very robust system for profiling where you add (copied from their docs) something like

var cpuprofile = flag.String("cpuprofile", "", "write cpu profile to `file`")
var memprofile = flag.String("memprofile", "", "write memory profile to `file`")

func main() {
    flag.Parse()
    if *cpuprofile != "" {
        f, err := os.Create(*cpuprofile)
        if err != nil {
            log.Fatal("could not create CPU profile: ", err)
        }
        defer f.Close() // error handling omitted for example
        if err := pprof.StartCPUProfile(f); err != nil {
            log.Fatal("could not start CPU profile: ", err)
        }
        defer pprof.StopCPUProfile()
    }

    // ... rest of the program ...

Have we tried adding something like that to the commands you want to profile?