Add the ability to execute notebooks via the CLI

[bt-maps]

Add options to cli to inject input parameters at runtime and allow parameterized livebooks to programmatically execute a workflow/broadway multi-stage data pipeline.

Basically, allow livebooks with inputs (or exported scripts) to be used as-is in multi-stage broadway pipelines and pass values downstream.

• similar to Papermill for Jupyter notebooks

Brent

[josevalim]

You can use inputs to parameterized Livebook and fill those in the UI. Because our inputs use the IO API, they also work when the notebooks run in the CLI (i.e. we will request the user to fill info in as it happens). So this functionality is already in place but in a way.

The biggest issue is that we don't have an option to execute notebooks via the CLI.

[jonatanklosko]

Depends on https://github.com/livebook-dev/kino/issues/54.

[houllette]

I was hoping to bump this issue as I'm currently working on a pretty novel idea using Livebooks in an educational manner and would like a way to execute notebooks via the CLI in order to automate grading code output. Any help would be appreciated! 😄

[jonatanklosko]

Hey @houllette, this is not a priority for us right now, but I put together a script for basic evaluation here. It needs to be run within the Livebook mix project:

git clone https://github.com/livebook-dev/livebook.git
cd livebook
mix dev.setup

# Download the script and use as:
mix run livebook_run.exs ./path/to/notebook.livemd

Hopefully this helps or gives you a starting point :)

[houllette]

Hey @jonatanklosko - Sorry to bother you again after so much time as elapsed since you wrote the aforementioned script. I was just starting to work with it more and am running into issues - I tried debugging it myself as best as possible but it appears I'm running into a wall.

I'm getting an error when the code enters the Livebook.Session.Data.apply_operation/2 function during your script execution on Line 38. The error I'm specifically seeing is 1st argument: not an atom.

At first I thought it was the .livemd file I was running the script on, but I've come to realize that it won't work even on just a super basic .livemd file that only has one code block returning an atom.

[jonatanklosko]

Hey @houllette! There was an internal change that broke it, I've just updated the script :)

[sonic182]

In jupyter there is nbconvert for notebooks to other formats (python scripts included of course) https://nbconvert.readthedocs.io/en/latest/index.html

It may be nice to have something like nbconvert in elixir for livebooks