Leading colon not suppressed when wrapping result block

[abradd]

When using the :wrap header argument the leading colon is not suppressed. Below I compare ob-python with emacs-jupyter:

#+begin_src python :results output :wrap org
  print('test')
#+end_src

#+RESULTS:
#+begin_src org
test
#+end_src

#+begin_src jupyter-python :session test :results output :wrap org
  print('test')
#+end_src

#+RESULTS:
#+begin_org
: test
#+end_org

Is it possible to suppress the leading colon when wrapping result blocks?

I also noticed this behavior when using:

#+begin_src jupyter-python :session test :results output org

[nnicandro]

Typically there shouldn't be a reason to supply header arguments like :wrap since Jupyter provides enough information to determine how results should be displayed.

Printed output of a source block normally does not contain valid org syntax which is why it is placed in either fixed-width (lines containing a leading colon) or example-block elements.

Do you mind explaining what you are trying to accomplish by using :wrap?

[dcherian]

Printed output of a source block normally does not contain valid org syntax which is why it is placed in either fixed-width (lines containing a leading colon) or example-block elements.

I've run into this problem trying to print a latex string to the org file, using pandas.DataFrame.to_latex() for example. My current workaround is to write to a file; and then include that file in org.

[nnicandro]

@dcherian Maybe the DataFrame gives a text/latex mimetype? Try returning the DataFrame object and supply a :display latex as a header argument. You can also create custom mime bundles using something like

from IPython.display import publish_display_data
publish_display_data({'text/latex': pandas.DataFrame.to_latex()});

And then to have the latex inserted directly instead of in an export block you can pass a :results raw header argument.

[dcherian]

Maybe the DataFrame gives a text/latex mimetype? Try returning the DataFrame object and supply a :display latex as a header argument.

This did not work AFAIR.

publish_display_data looks interesting. This should be in the README.

In general though if :wrap is specified, I think emacs-jupyter should just take the string and wrap it.

[nnicandro]

Actually there is a simpler method for Python to return Latex

from IPython.core.display import Latex
Latex(pandas.DataFrame.to_latex())

OK, back to this issue...

The problem with just wrapping output in the block specified by :wrap is that we have to handle both printed output on stdout as well as result based output like plot images which can appear interspersed with stdout messages.

In Jupyter there are three message types that need to be handled:

• stream (as in a stdout stream)
• display_data (as in display plot image data)
• execute_result (essentially the same as display_data but considered as the result of a code block)

So if we specify :wrap type as meaning to wrap stream results in a type block what do we do if evaluation of a code block produces a message list like stream, display_data, stream? Should we also wrap the display_data contents in the type block? I think it depends on type.

For example, if type was org, we could process the display_data message normally and place it in the block. On the other hand, suppose the display_data message contained an image and type was markdown. It wouldn't make sense to process the display_data message into an org file link. If we want to handle it properly, we end up having to translate the file link into markdown. This means we would have to specially handle the various values of type which doesn't seem like the right solution since type can be anything.

A possible solution would be to use org-export. For example, finding an export backend based on type, converting the display_data message into its org-mode representation and use org-export to convert into type.

Or we can just go with the strategy of just handling display_data normally and wrapping it in a type block along with everything else. If a user specifies :wrap they know what the code block produces anyways. I'm not sure about this solution either.

[dcherian]

Or we can just go with the strategy of just handling display_data normally and wrapping it in a type block along with everything else. If a user specifies :wrap they know what the code block produces anyways.

I think this is a fair assumption or at least it seems like the simplest way forward to get reasonable default behaviour. You could raise an error if :wrap is specified and there is a display_data message?

[abradd]

Do you mind explaining what you are trying to accomplish by using :wrap?

I originally did it to remove the leading colon in ob-ipython so that the result would appear as if it were part of the prose in a notebook as opposed to some type of example block.

It also enables a workflow where I can prepend #+caption, #+name and attributes to an image as in John Kitchin's post.

Some recent work I was doing looked like:

"Some text about an equivalent circuit that looks like the function below (which i generate with sympy in a code block)"

#+begin_src jupyter-python :session session-name :results output scalar raw :wrap org :exports results 

some code that generates latex output that I print to the results block

#+end_src

#+RESULTS:
#+begin_org
 $$Z_{cap} = \frac{i^{\alpha} C_{in} Rp Rs Y \omega \omega^{\alpha} - i i^{\alpha} Rp Y \omega^{\alpha} + C_{in} Rp \omega + C_{in} Rs \omega - i}{i^{\alpha} C_{in} Rp Y \omega \omega^{\alpha} + C_{in} \omega}$$ 
 
  and 
 
  $$Z_{nocap} = \frac{i^{\alpha} Rp Rs Y \omega^{\alpha} + Rp + Rs}{i^{\alpha} Rp Y \omega^{\alpha} + 1}$$
#+end_org

when I export the output to html the code block is ignored and the latex equations are rendered inline.

For image outputs a block looks like:

#+begin_src jupyter-python :session session-name :results raw :file "file-path" :wrap (src-decorate "name" nil "caption" '("#+attr_org :width 1000")) :exports results

some computation that produces a figure

#+end_src

#+RESULTS:
#+begin_org
#+name: name
#+caption: caption
#+attr_org :width 1000
[[file:filepath]]
#+end_org

and again the output will be inline when exported. This was what I used with ob-ipython anyway. There may be an easier solution with emacs-jupyter and I'm open to suggestions. In essence I am attempting to create a document that when exported can present as though there were no source blocks while allowing me to label and reference figures and equations etc.

The problem with just wrapping output in the block specified by :wrap is that we have to handle both printed output on stdout as well as result based output like plot images which can appear interspersed with stdout messages.

I've seen this behavior with ob-ipython, but haven't seen the issue with jupyter-python yet. If I recall correctly it would happen with ob-ipython when I defined details like the title after populating the axis with the data. At the moment the final step in a block that returns display_data with jupyter-python is plt.show() which seems to work fine as you can see in the above example.

Or we can just go with the strategy of just handling display_data normally and wrapping it in a type block along with everything else. If a user specifies :wrap they know what the code block produces anyways. I'm not sure about this solution either.

I think its reasonable for the user to take responsibility for what they put inside a :wrap block. In particular I don't think the use of :wrap implies further translation of the output by emacs-jupyter.

[johnbcoughlin]

First of all, fantastic package, I'm using it for Julia instead of the jupyter web interface and so far it's great.

I am experiencing roughly the same issue -- I am generating a table of results in code like so:

  using Printf

  @printf("| k |         3rd Order Runge Kutta\n ")
  for j=2:p
      @printf("| %f | %0.4f |\n", ks[j], data_rk[j-1]/data_rk[j])
  end

Ideally, this code block would output a string which I could insert directly into the org buffer with :results value raw. Instead, the leading colons are not stripped:

#+RESULTS:
: | k |         3rd Order Runge Kutta
:  | 0.005000 | 8.3641 |
: | 0.002500 | 8.1877 |
: | 0.001250 | 8.0954 |

The documentation suggests that an alternative is to return an array object, but frankly, using printf is actually easier than constructing an array containing the appropriate headers and so on. Moreover, printing an array return value as a table doesn't seem to quite work for Julia. There's an n x n implementation problem here, to get everything to "just work"--the package would have to understand how every supported language outputs every supported object type. A workaround where we can use our programming language of choice to generate org-mode compatible syntax seems like a reasonable thing.

edit: here is the full header line of my src block:

#+begin_src jupyter-julia :exports both :results value raw :async yes

Although I just tested and the behavior is identical with :async no as well.

[fleimgruber]

@abradd are you still using your approach mentioned in https://github.com/nnicandro/emacs-jupyter/issues/105#issuecomment-489273032? I was trying out your approach, but always get something like this

#+RESULTS:
[[file:filepath]]
#+begin_org
#+name: name
#+caption: caption
#+attr_org :width 1000
#+end_org

whenever I evaluate the src block without an existing #+RESULTS block. If there exists a #+RESULTS block already, i.e. running the source block twice, I get this

#+RESULTS:
[[file:filepath]]
#+begin_org
#+name: name
#+caption: caption
#+attr_org :width 1000
#+end_org
#+begin_org
#+name: name
#+caption: caption
#+attr_org :width 1000
#+end_org

any ideas?

[abradd]

I haven't worked with the source blocks in over 6 months. So something may have changed since.

What are the src block options you are using to capture the file?

I have used :file in the past and it worked.

[fleimgruber]

What are the src block options you are using to capture the file?

The src block header reads as

#+begin_src julia :results raw :file "filepath.png" :wrap (src-decorate "name" "caption" '("#+attr_org: :width 400")) :exports results

[fleimgruber]

FYI, for a default width I am now using org-image-actual-width with a value of '(640) as described e.g. in https://github.com/nnicandro/emacs-jupyter/issues/178#issuecomment-530070846. This is what I would use #+ATTR_ORG: for most of the time. Still, it would be nice if the user could override this on a per source block basis as outlined in this issue.

[nnicandro]

I agree that in some cases it makes sense to just insert whatever the kernel sends as is when :wrap is specified, for example when printing to stdout. But what about all the other result types that can be sent by a kernel? What do we do, for example, when a kernel sends an image and :wrap is specified? Or what if it sends markdown or html or whatever is currently wrapped in an export block when sent? In those cases it isn't entirely clear what should be done. Should we just insert the content as is without any Org formatting? That could make sense for markdown, html, and the like but doesn't really make sense for an image.

In the case of :results raw I agree that we should handle that header argument in the way that standard Org does but, again, what should be done with all of the result types that can be sent by a kernel? If the meaning of :results raw in the case of Jupyter source blocks is whatever the kernel sends is interpreted as raw Org mode syntax, then it makes sense to do what we do now for all cases except stdout. In the case of stdout we do special formatting (inserting :), instead we can just insert stdout as is assuming it to be raw Org mode syntax.