live printing in notebook

[germayneng]

Hi!

Thanks for creating such a great package. In my notebook, for example if i am training a ML model, typically in the notebook, the training process will output a verbose each step. But if i were to run using papermill, the particular cell will only show the output only after the cell is done executing. Is there any options for CLI to output the verbose message live rather than after the cell is done running?

e.g

training round 1...
training round 2.. 

...etc

[MSeal]

Yes. You can use --log-output and it will stream the logs to the command line live. To note, older versions of nbconvert won't live log, so make sure you have nbconvert >=5.6

[SultanOrazbayev]

When I ran a notebook from within a python script using:

papermill.execute_notebook(input_path=nbk_path, output_path=nbk_log_path, parameters=myparameters, kernel_name='python3', log_output=True, autosave_cell_every=60)

I don't see the output in my shell and also I don't see the print results of a cell that is still running (I was hoping this would be achieved by the autosave every 60 minutes). Is this expected when executing the notebook from within another script/notebook?

[MSeal]

So when setting log_output in a python script it's making python log the output to the logging.getLogger('papermill'). This lets you control how the log message propegates e.g.:

logging.basicConfig(level='INFO', format="%(message)s")

would setup your default logging experience to print to stderr.

Another option is to set the stdout_file and/or stderr_file which additional writes if set:

result = pm.execute_notebook(
    input_path=nbk_path,
    output_path=nbk_log_path,
    parameters=myparameters,
    kernel_name='python3', 
    log_output=True,
    autosave_cell_every=60,
    stdout_file=sys.stdout,
    stderr_file=sys.stderr,
)

As to the autosave, that feature saves the output_path notebook in a regular pattern so you can view the results before a cell is completed, rather than having anything to do with the logging of the process.

[SultanOrazbayev]

This is great, thank you!