Support for executing notebooks in parallel during JupyterBook Build

[nishikantparmariam]

Context

If there are big and many notebooks, then jb build . can take too much time. Specially when building notebooks is a part of a recurring automated process or a CI pipeline. This may be improved by executing notebooks in parallel.

There should be a way to specify the number of notebooks to be executed in parallel (default 1) or the number of job that can be used to execute the notebooks, say using jb build . -j 5. I could not find a way to do this currently.

Please let me know if there is an existing solution / update / workaround for this. Else, I would like to work on PR as per below proposal.

Proposal

There is a similar issue - issues#1719

Based on initial findings -

• MyST-NB already supports parallel execution as per this changelog.
• Introducing an optional argument j or job, usage - jb build . -j 5.
• build_sphinx already supports the job argument, hence we can pass the job argument from here as well.

Tasks and updates

No response

[migueltorrescosta]

Temporary solution: https://github.com/executablebooks/jupyter-cache

Just started using jupyter-book ( amazing project :raised_hands: ), and this feature would make it a lot better: iterating on the web generated pages locally is very slow due to this constraint. It is possible to work around them with notebook_execute: off with nbconvert, or by only including the notebook I am currently working on, however no solution is as smooth as allowing the parallel execution of notebooks.

Is there something I can do to help with this feature?

[bsipocz]

While this issue was open for the JB1 stack, it's still a relevant feature request for JB2 (though also tracked in mystmd in https://github.com/jupyter-book/mystmd/issues/1831)