doit icon indicating copy to clipboard operation
doit copied to clipboard

num_process = num_cpus

Open jakajancar opened this issue 3 years ago • 6 comments

It would be nice if doit could use as many processes as there are CPUs instead of only a hardcoded number.

Fund with Polar

jakajancar avatar May 20 '21 13:05 jakajancar

How should be the command line ? -n cpu ?

schettino72 avatar May 21 '21 03:05 schettino72

Seems reasonable.

Btw, the default is now 0, which is silly, you can’t do anything with 0 cpus :) it should be changed to 1 and 0 should be invalid.

Or 0 or -1 could also mean #cpus if for some reason you want the type to always be int.

jakajancar avatar May 21 '21 03:05 jakajancar

Btw, the default is now 0, which is silly, you can’t do anything with 0 cpus :) it should be changed to 1 and 0 should be invalid.

You are right, it is silly but it actually has different meaning.

  • 0 means do not use any parallel execution - tasks will be executed in same process as doit main process (control/scheduler)
  • 1 means one extra process will be used to execute tasks

So I guess what really should be done is:

  • not specifying -n would have the value of zero
  • specifying -n without a number, use number of CPUs

So the option should be a combination of acting like a flag and taking a value... Not sure the correct terms but "default" seems ambiguous for these 2 scenarios.

I guess doit command parsing currently is not able to handle this. Note doit uses its own command parsing based getopt so should be feasible to implement this. Not sure how other libs handle this use-case.

schettino72 avatar May 21 '21 10:05 schettino72

argsparse: https://stackoverflow.com/a/15301183/64444

schettino72 avatar May 21 '21 10:05 schettino72

click: https://github.com/pallets/click/issues/549 I cant tell if this feature was implemented or not or how to use it...

schettino72 avatar May 21 '21 10:05 schettino72

Ah, didn't know 0 is different than 1, makes sense!

The flag/value combo parameter is unusual, I don't remember seeing it before.

I think your initial "-n cpu" idea is best!

jakajancar avatar May 21 '21 17:05 jakajancar