jupyter icon indicating copy to clipboard operation
jupyter copied to clipboard

First row gets right-shifted when table returned from jupyter-python src block

Open xor-xor opened this issue 3 years ago • 2 comments

#+begin_src jupyter-python :kernel test :session py :results table
[[1, 99], [2, 88], [3, 77]]
#+end_src

#+RESULTS:
|   |  1 | 99 |
| 2 | 88 |    |
| 3 | 77 |    |

vs.

#+begin_src python :results table
return [[1, 99], [2, 88], [3, 77]]
#+end_src

#+RESULTS:
| 1 | 99 |
| 2 | 88 |
| 3 | 77 |

Logs after setting (setq jupyter--debug t):

executing Jupyter-Python code block...
SENDING: :execute-request 1d9a076e-b6ce-4954-8e99-9618fbfe717c (:code [[1, 99], [2, 88], [3, 77]] :silent :json-false :store_history t :user_expressions #s(hash-table size 1 test eql rehash-size 1.5 rehash-threshold 0.8125 data ()) :allow_stdin t :stop_on_error :json-false)
SENT: (:shell 1d9a076e-b6ce-4954-8e99-9618fbfe717c)
MESSAGE: (:iopub :status (:execution_state busy))
RUN-HOOK: jupyter-iopub-message-hook
MESSAGE: (:iopub :execute-input (:code [[1, 99], [2, 88], [3, 77]] :execution_count 3))
RUN-HOOK: jupyter-iopub-message-hook
MESSAGE: (:iopub :execute-result (:data (:text/plain [[1, 99], [2, 88], [3, 77]]) :metadata nil :execution_count 3))
RUN-HOOK: jupyter-iopub-message-hook
MESSAGE: (:iopub :status (:execution_state idle))
RUN-HOOK: jupyter-iopub-message-hook
DROPPING-REQ: c2a4808d-a13c-4396-8459-40630ba1ccd0
MESSAGE: (:shell :execute-reply (:status ok :execution_count 3 :user_expressions nil :payload []))
RUN-HOOK: jupyter-shell-message-hook
Code block evaluation complete. [2 times]

This happens both on 0.8.2 and 20210422.1451 versions of emacs-jupyter.

Reproduced on clean run of Emacs 27.2 (emacs-mac), i.e. without any irrelevant packages installed (only emacs-jupyter and its dependencies) and with ~/.emacs containing only:

(with-eval-after-load 'org
  (require 'org-tempo)
  (org-babel-do-load-languages 'org-babel-load-languages
                               '((python . t)
				 (jupyter . t)))
  (add-to-list 'org-structure-template-alist '("jp" . "src jupyter-python")))

(plus some standard settings related to MELPA, which are irrelevant here)

xor-xor avatar Nov 10 '21 19:11 xor-xor

The behaviour reported in this issue also happens on Emacs 26.3, which has Org 9.1.9 (btw, Emacs 27.2 has Org 9.4.4).

Regardless of the Emacs/Org version, this problem can be traced to orgtbl-to-generic function in org-table.el file (org-ctrl-c-ctrl-c -> org-babel-execute-src-block -> org-babel-insert-result -> orgtbl-to-orgtbl -> orgtbl-to-generic). But there's one weird thing happening here, which becomes more apparent, when we use slightly different examples:

#+begin_src python :results table
return [[1, 11], [2, 12], [3, 13]]
#+end_src

In the above, orgtbl-to-generic gets called once, with ((1 11) (2 12) (3 13)) as its table argument, and produces correct/expected output.

#+begin_src jupyter-python :kernel python3 :session test :results table
[[4, 44], [5, 55], [6, 66]]
#+end_src

This time, orgtbl-to-generic gets called twice - with the first call, table argument has similar form as in the previous example:

((4 44) (5 55) (6 66))

...but with the second one, it looks quite different:

((#("| 4 | 44 |\n| 5 | 55 |\n| 6 | 66 |\n" 0 1 (jupyter-org t org-table t))))

...and this second form gets corrupted somewhere in the guts of orgtbl-to-generic, producing incorrect result, and this result is inserted into the buffer.

So the question remains - why there are two different calls instead of just the first one?

xor-xor avatar Nov 11 '21 18:11 xor-xor

There are two different calls because the first one comes from jupyter-org-scalar function and the second one comes from Org itself. But the docstring of this function probably reveals the source of the problem: If VALUE is a list and can be represented as a table, return an `org-mode' table as a string. - why such conversion is needed? In case of org-babel-execute:python such list is returned without any conversion, as Org knows how to handle it (and probably this is what it expects, but I haven't found any docs on this, so I'm not sure).

xor-xor avatar Nov 14 '21 17:11 xor-xor