mage icon indicating copy to clipboard operation
mage copied to clipboard

Add helper for parallelisation limits

Open kortschak opened this issue 8 years ago • 3 comments

The code is trivial, but it might be nice to add a helper to provide parallelisation limits.

kortschak avatar Oct 03 '17 22:10 kortschak

I'm curious what problem this solves? I try to keep the ux as simple as possible.

natefinch avatar Oct 03 '17 23:10 natefinch

It is common in make-like tools to be able to specify a parallelisation limit/level, e.g. make -j N. mage already allows concurrent processing purely by virtue of being Go, but say for example the user's rules involved processing thousands of files, the naive approach would suffer significantly unless there were a limit on the number of concurrent goroutines running.

The user can write

var n = flag.Int("j", 0, "specify the number of jobs to run simultaneously")

type limiter struct {
    token chan struct{}
    wg    sync.WaitGroup
}

var limit = limiter{token: make(chan struct{}, n) }

func acquire() {
    limit.wg.Add(1)
    limit.token <- struct{}{}
}

func release() {
    <-limit.token
    limit.wg.Done
}

for each set of mage files they want to use concurrent processing in which this is an issue, but then you can say that about any specific problem/solution.

kortschak avatar Oct 04 '17 00:10 kortschak

ahh, thanks for the more detailed explanation.

natefinch avatar Oct 04 '17 00:10 natefinch