jupyter-cache icon indicating copy to clipboard operation
jupyter-cache copied to clipboard

Alllow for consecutive (depedent) notebook executions

Open chrisjsewell opened this issue 5 years ago • 1 comments

Although this would introduce some extra complexity, it is (I think) a current requirement of Jupinx.

Once #10 is implemented it would be possible to 'orchestrate' consecutive notebook execution, whereby:

  1. A notebook is executed and its final state and artefacts committed.
  2. A subsequent notebook is staged for execution with (1)'s nbfile and/or artefacts as it's assets.
  3. That notebook is executed

The orchestration could possibly be done at the staging phase, from a top-level; stating which notebook relied on which, or from the notebook-level; stating a relative path dependency in the metadata (this may be less robust). Then the cache or executor would work out a 'branching tree' of execution paths (checking it is acyclic) and decide how to execute appropriately.

The artefacts from the first notebook could be either/or files that the notebook outputs or its final kernel state (using something like dill.dump_session)

chrisjsewell avatar Feb 25 '20 06:02 chrisjsewell

FYI, this come up from discussion with @mmcky, @choldgraf may also be interested

chrisjsewell avatar Feb 25 '20 06:02 chrisjsewell