papermill icon indicating copy to clipboard operation
papermill copied to clipboard

Implement bash translator

Open kdm9 opened this issue 2 years ago • 2 comments

Support translation of parameters to bash/*NIX shell

Thanks for papermill! I have used it for python notebooks, but a lot of my work is best done in shell, for which I use jupyter notebooks with the bash_kernel (via *.sh scripts and jupytext).

Alas papermill doesn't seem to support bash/shell script, due to a hitherto unimplemented bash translator. This PR implements a BashTranslator, and therefore enables bash kernel notebooks to be parametrised. This code supports str, list, bool[^1], None, int, and float parameter values. Dict could in theory be supported via associative arrays but for now they are not implemented[^2].

For a functioning example, please see the attached notebook.

Best, K

[^1]: Bash doesn't really have a bool type, but true and false are at least common and sensible string constants for truthy-ness and falsy-ness. See for example this SO answer. [^2]: Associative arrays require quite recent bash versions, and I've never seen them used outside tutorials. Supporting them would also likely require a more involved code path as one must pre-declare an associative array in bash, and then declare one entry per line (see link above).

kdm9 avatar Jul 03 '22 17:07 kdm9

Alas GH doesn't support attaching *.ipynb, so I have had to rename it *.txt, but here's the test notebook.

test.ipynb.txt

The following params.yml can be used with this:

say: "Write a book"
people:
  - Alice
  - Bob
  - Robert'); DROP TABLE students;-- # XKCD's favourite son

kdm9 avatar Jul 03 '22 18:07 kdm9

A friendly ping to the nteract team about this PR. All tests pass locally, and this is currently blocking us from using papermill in our analyses.

kdm9 avatar Aug 03 '22 11:08 kdm9

@rohitsanj tests implemented and passing. There are some unrelated failures on my machine, all to do with the google cloud interaction, which I assume is because I don't have a GCS environment correctly configured. Any chance you can allow the CI to run on this PR, so we can see if all is green in a well-configured environment?

kdm9 avatar Aug 13 '22 09:08 kdm9

Codecov Report

Merging #674 (000a80a) into main (98013f0) will increase coverage by 0.13%. The diff coverage is 90.90%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #674      +/-   ##
==========================================
+ Coverage   91.72%   91.86%   +0.13%     
==========================================
  Files          17       17              
  Lines        1583     1622      +39     
==========================================
+ Hits         1452     1490      +38     
- Misses        131      132       +1     
Impacted Files Coverage Δ
papermill/translators.py 98.38% <90.90%> (-0.48%) :arrow_down:
papermill/parameterize.py 97.91% <0.00%> (-0.05%) :arrow_down:
papermill/execute.py 100.00% <0.00%> (ø)
papermill/engines.py 98.37% <0.00%> (+0.09%) :arrow_up:
papermill/iorw.py 81.15% <0.00%> (+0.91%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 98013f0...000a80a. Read the comment docs.

codecov[bot] avatar Aug 13 '22 11:08 codecov[bot]