Light file format capable of encoding code cell output?

[tribbloid]

Just a hypothesis on your the design of the light format:

If code cell output can be memorised in any of the light file format (*.lgt.py or *.lgt.gl) as a special comment, then there is no need to keep ipynb file anymore, as all the information are already in the source code, which will be parsed in jupytext, but ignored in most other IDE (actually, I think other IDE will gradually start to support it if it becomes a standard)

[mwouts]

Hello @tribbloid , thanks for discussing this. This is an interesting point. I have seen one tentative in this direction in the notedown project, but I have not tried it personally.

I am curious of ways to represent outputs in the text file, but I'd like to preserve the below:

• users like the small size of the text representation (files may inflate if outputs are included)
• they also like the absence of noise for version control

We could think of representing plots as images. But we would have to keep track of all the images, and delete removed plots.... And then we have other types of outputs, like JS libraries or plots, or simpler, Pandas tables... How do you suggest that we represent a large HTML table?

[goerz]

For the markdown format, it might be nice to have an option in the jupytext command line utility to embed the output in the exact same way that notedown has it: as a json block with the "raw" output data. In the conversion back to ipynb, a json block that comes after a python block could be inserted into the ipynb as the output for the preceding input cell (assuming it parses as valid data).

This could be a way for jupytext.vim to allow editing of input data while preserving the output (which is a debatable thing to do, but notedown allows for it, and it could be helpful with things like cleaning up whitespace in input with vim). In general, though, I also like the "small size of the text representation", so this should definitely be optional.

Not sure how to include it in the python format; I guess a comment block with the raw json data?

[mwouts]

Hello @goerz, I agree that the user may not want to lose the outputs when he just edits the inputs - just like in Jupyter.

Adding outputs to the text representation is a major change and I am not ready yet to experiment this. However, there is something else that we could do: we could improve the way that former outputs are distributed to the new inputs.

In the current implementation, the outputs cells are distributed in order to cells with matching input (up to blank line changes). We could try to improve the matching algorithm and distribute unmatched outputs to cells with modified inputs. Do you think that would fit to your use case?

[goerz]

I actually don't think I care about that feature that much, personally. I just wanted to give my 2 cents on how it might look like if you did decide to implement it. Definitely no rush!

[mwouts]

@tribbloid , I am considering including outputs in the Markdown format. Do you want to have a look at #220 ? Thanks

[zippeurfou]

@mwouts just a follow up on this one. I am a new user and I specifically installed jupytext for this. I was hoping doing: jupytext --to md my_file.py --execute would convert my python light script into a markdown with the output. It seems like it did execute the file but the markdown does not display output. To be frank this would have been fantastic as it would allow me to just not have to use jupyter at all anymore and just use the light format and have a hook that generate the markdown for me for my report. Can you think of any alternative/way I could make it work? Best,

[mwouts]

@zippeurfou you could pipe an ipynb file generated by Jupytext into jupyter nbconvert like this:

echo "1+1" | jupytext --from py --to ipynb | jupyter nbconvert --stdin --to markdown --execute --stdout

which results in:

[NbConvertApp] Converting notebook into markdown
```python
1+1
```




    2

Obviously you can also intermediate files, e.g.

jupytext my_file.py --to ipynb
jupyter nbconvert my_file;ipynb --to markdown --execute

[zippeurfou]

Thanks so much this does the trick :). Might be nice to put it in a FAQ/doc as I think this creates a really nice use case where you can do your EDA straight in python and get a markdown with your analysis. PS: It's a bit funny that this workflow tries to avoid using a notebook but end up using it for the conversion ahaha :)