elpy-goto-definition breaks on emacs 27.1 + python 3.9.1

[akshaybadola]

Summary

elpy-goto-definition was giving nil results on any function on emacs 27.1 with python 3.9.1

Steps to reproduce

Call elpy-goto-defintion with emacs 27.1 and python 3.9.1 on any function. Upon a quick debug I saw the error from elpy-rpc as

((message . "Object of type PosixPath is not JSON serializable")
 (code . 500)
 (data
  (traceback . "Traceback (most recent call last):
  File \"/home/joe/.emacs.d/elpa/elpy-20201115.1811/elpy/rpc.py\", line 104, in handle_request
    self.write_json(result=result,
  File \"/home/joe/.emacs.d/elpa/elpy-20201115.1811/elpy/rpc.py\", line 77, in write_json
    self.stdout.write(json.dumps(kwargs) + \"\\n\")
  File \"/lib64/python3.8/json/__init__.py\", line 231, in dumps
    return _default_encoder.encode(obj)
  File \"/lib64/python3.8/json/encoder.py\", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File \"/lib64/python3.8/json/encoder.py\", line 257, in iterencode
    return _iterencode(o, 0)
  File \"/lib64/python3.8/json/encoder.py\", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type PosixPath is not JSON serializable
")))

A simple fix in rpc.py like this works:

    def json_defaults(self, x):
        if isinstance(x, Path):
            return str(x)

    def write_json(self, **kwargs):
        """Write an JSON object on a single line.

        The keyword arguments are interpreted as a single JSON object.
        It's not possible with this method to write non-objects.

        """
        self.stdout.write(json.dumps(kwargs, default=self.json_defaults) + "\n")
        self.stdout.flush()

[tuedachu]

It works for me! Thanks.

[sten0]

Thank you for your work on this @akshaybadola. I tried it just now, hoping it would be enough to solve failing elpy-multiedit-python-symbol-at-point-should-save-some-buffers on Debian sid (Emacs 27.1, Python 3.9.1), but unfortunately

--- a/elpy/rpc.py
+++ b/elpy/rpc.py
@@ -67,6 +67,10 @@ class JSONRPCServer(object):
             raise EOFError()
         return json.loads(line)
 
+    def json_defaults(self, x):
+        if isinstance(x, Path):
+            return str(x)
+
     def write_json(self, **kwargs):
         """Write an JSON object on a single line.
 
@@ -74,7 +78,7 @@ class JSONRPCServer(object):
         It's not possible with this method to write non-objects.
 
         """
-        self.stdout.write(json.dumps(kwargs) + "\n")
+        self.stdout.write(json.dumps(kwargs, default=self.json_defaults) + "\n")
         self.stdout.flush()
 
     def handle_request(self):

still results in

Test elpy-multiedit-python-symbol-at-point-should-save-some-buffers backtrace:
  signal(error ("Selecting deleted buffer"))
  error("Selecting deleted buffer")
  (if (and (eq (car err) 'error) (stringp (car (cdr err))) (string-match "not implemented" (car (cdr err)))) 'not-supported (error (car (cdr err))))
  (condition-case err (elpy-rpc-get-usages) (error (if (and (eq (car err) 'error) (stringp (car (cdr err))) (string-match "not implemented" (car (cdr err)))) 'not-supported (error (car (cdr err))))))
  (let ((usages (condition-case err (elpy-rpc-get-usages) (error (if (and (eq ... ...) (stringp ...) (string-match "not implemented" ...)) 'not-supported (error (car ...))))))) (cond ((eq usages 'not-supported) (call-interactively 'elpy-multiedit) (message (concat "Using syntactic editing " "as current backend does not support get_usages."))) ((null usages) (call-interactively 'elpy-multiedit) (if elpy-multiedit-overlays (message (concat "Using syntactic editing as no usages of the " "symbol at 
  (if (or elpy-multiedit-overlays use-symbol-p (use-region-p)) (call-interactively 'elpy-multiedit) (save-some-buffers) (let ((usages (condition-case err (elpy-rpc-get-usages) (error (if (and ... ... ...) 'not-supported (error ...)))))) (cond ((eq usages 'not-supported) (call-interactively 'elpy-multiedit) (message (concat "Using syntactic editing " "as current backend does not support get_usages."))) ((null usages) (call-interactively 'elpy-multiedit) (if elpy-multiedit-overlays (message (conca
  elpy-multiedit-python-symbol-at-point()
  (progn (fset 'save-some-buffers vnew) (elpy-multiedit-python-symbol-at-point) (let ((value-6060 (gensym "ert-form-evaluation-aborted-"))) (let (form-description-6061) (if (unwind-protect (setq value-6060 save-some-buffers-called) (setq form-description-6061 (list '... :form 'save-some-buffers-called :value value-6060)) (ert--signal-should-execution form-description-6061)) nil (ert-fail form-description-6061))) value-6060))
  (unwind-protect (progn (fset 'save-some-buffers vnew) (elpy-multiedit-python-symbol-at-point) (let ((value-6060 (gensym "ert-form-evaluation-aborted-"))) (let (form-description-6061) (if (unwind-protect (setq value-6060 save-some-buffers-called) (setq form-description-6061 (list ... :form ... :value value-6060)) (ert--signal-should-execution form-description-6061)) nil (ert-fail form-description-6061))) value-6060)) (fset 'save-some-buffers old))
  (let* ((vnew #'(lambda nil (setq save-some-buffers-called t))) (old (symbol-function 'save-some-buffers))) (unwind-protect (progn (fset 'save-some-buffers vnew) (elpy-multiedit-python-symbol-at-point) (let ((value-6060 (gensym "ert-form-evaluation-aborted-"))) (let (form-description-6061) (if (unwind-protect (setq value-6060 save-some-buffers-called) (setq form-description-6061 ...) (ert--signal-should-execution form-description-6061)) nil (ert-fail form-description-6061))) value-6060)) (fset 
  (let ((save-some-buffers-called nil)) (let* ((vnew #'(lambda nil (setq save-some-buffers-called t))) (old (symbol-function 'save-some-buffers))) (unwind-protect (progn (fset 'save-some-buffers vnew) (elpy-multiedit-python-symbol-at-point) (let ((value-6060 (gensym "ert-form-evaluation-aborted-"))) (let (form-description-6061) (if (unwind-protect ... ... ...) nil (ert-fail form-description-6061))) value-6060)) (fset 'save-some-buffers old))))
  (progn (let ((save-some-buffers-called nil)) (let* ((vnew #'(lambda nil (setq save-some-buffers-called t))) (old (symbol-function 'save-some-buffers))) (unwind-protect (progn (fset 'save-some-buffers vnew) (elpy-multiedit-python-symbol-at-point) (let ((value-6060 ...)) (let (form-description-6061) (if ... nil ...)) value-6060)) (fset 'save-some-buffers old)))))
  (progn (setq elpy-rpc-timeout 200) (progn (let ((save-some-buffers-called nil)) (let* ((vnew #'(lambda nil ...)) (old (symbol-function 'save-some-buffers))) (unwind-protect (progn (fset 'save-some-buffers vnew) (elpy-multiedit-python-symbol-at-point) (let (...) (let ... ...) value-6060)) (fset 'save-some-buffers old))))))
  (unwind-protect (progn (setq elpy-rpc-timeout 200) (progn (let ((save-some-buffers-called nil)) (let* ((vnew #'...) (old (symbol-function ...))) (unwind-protect (progn (fset ... vnew) (elpy-multiedit-python-symbol-at-point) (let ... ... value-6060)) (fset 'save-some-buffers old)))))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (setq elpy-rpc-timeout 200) (progn (let ((save-some-buffers-called nil)) (let* ((vnew ...) (old ...)) (unwind-protect (progn ... ... ...) (fset ... old)))))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))
  (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (setq elpy-rpc-timeout 200) (progn (let ((save-some-buffers-called nil)) (let* (... ...) (unwind-protect ... ...))))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))))
  (progn (if elpy-enabled-p (progn (message "Elpy was not deactivated by the previous test, dea...") (elpy-disable))) (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (setq elpy-rpc-timeout 200) (progn (let (...) (let* ... ...)))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))) (if (and (boundp 'elpy-enabled-p) elpy-enabled-p) (progn (elpy-disable))))
  (unwind-protect (progn (if elpy-enabled-p (progn (message "Elpy was not deactivated by the previous test, dea...") (elpy-disable))) (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (setq elpy-rpc-timeout 200) (progn (let ... ...))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))) (if (and (boundp 'elpy-enabled-p) elpy-enabled-p) (progn (elpy-disable)))) (let ((--dolist-tail-- (process-list)) proc) (while --dol
  (let ((old-process-list (process-list)) (old-buffer-list (buffer-list))) (unwind-protect (progn (if elpy-enabled-p (progn (message "Elpy was not deactivated by the previous test, dea...") (elpy-disable))) (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (setq elpy-rpc-timeout 200) (progn ...)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))) (if (and (boundp 'elpy-enabled-p) elpy-enabled-p) (progn (elpy-disabl
  (lambda nil (let ((old-process-list (process-list)) (old-buffer-list (buffer-list))) (unwind-protect (progn (if elpy-enabled-p (progn (message "Elpy was not deactivated by the previous test, dea...") (elpy-disable))) (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn ... ...) (and ... ...)))) (if (and (boundp 'elpy-enabled-p) elpy-enabled-p) (progn (elpy-disable)))) (let ((--dolist-tail-- (process-list)) proc) (while --doli
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test :name elpy-multiedit-python-symbol-at-point-should-save-some-buffers :documentation nil :body (lambda nil ...) :most-recent-result #s(ert-test-failed :messages "" :should-forms nil :duration 0.000568159 :condition ... :backtrace ... :infos nil) :expected-result-type :passed :tags nil) :result #s(ert-test-failed :messages "" :should-forms nil :duration 0.000568159 :condition (error "Selecting deleted buffer") :backtrace (... .
  ert-run-test(#s(ert-test :name elpy-multiedit-python-symbol-at-point-should-save-some-buffers :documentation nil :body (lambda nil (let ((old-process-list ...) (old-buffer-list ...)) (unwind-protect (progn ... ... ...) (let ... ...) (let ... ...)))) :most-recent-result #s(ert-test-failed :messages "" :should-forms nil :duration 0.000568159 :condition (error "Selecting deleted buffer") :backtrace (#s(backtrace-frame :evald t :fun signal :args (error ...) :flags nil :locals (... ... ... ...) :bu
  ert-run-or-rerun-test(#s(ert--stats :selector t :tests ... :test-map #<hash-table eql 465/465 0x156ddd54fac9> :test-results ... :test-start-times ... :test-end-times ... :passed-expected 189 :passed-unexpected 0 :failed-expected 0 :failed-unexpected 1 :skipped 0 :start-time ... :end-time nil :aborted-p nil ...) #s(ert-test :name elpy-multiedit-python-symbol-at-point-should-save-some-buffers :documentation nil :body ... :most-recent-result ... :expected-result-type :passed :tags nil) #f(compile

@tuedachu, are you reporting success with 1.35.0 or 4032c72? @akshaybadola, I wonder if elpy-multiedit-python-symbol-at-point (elpy.el:~2450) might need a compat fix too? If you could take a look at this backtrace and that function, and possibly test/elpy-multiedit-python-symbol-at-point-test.el I'd really appreciate it :-)

[akshaybadola]

Hi @sten0, I'm actually avoding python 3.9 for most projects right now and I didn't really run the tests for the whole elpy package. In that particular instance it was because of an attempted implicit conversion from pathlib.Path to str.

I haven't used elpy-multiedit or elpy-refactor that much. My guess is that "selecting deleted buffer" would probably result from elpy trying to access some buffer that got killed. I can take a look and see what comes out.

BTW, I ran all elpy-multiedit- tests with emacs 27.1 and python 3.9.1 with jedi==0.18.0 and got no errors. I'm using a Fedora 33 though.

Further edit:

I remember running into this error earlier and I had patched the function as following. Perhaps it might help.

(defun elpy-multiedit-stop ()
  "Stop editing multiple places at once."
  (interactive)
  (dolist (ov elpy-multiedit-overlays)
    (delete-overlay ov))
  (setq elpy-multiedit-overlays nil)
  (if (get-buffer "*Elpy Edit Usages*")
      (kill-buffer "*Elpy Edit Usages*")))

[xdxu]

The patch doesn't work for me. I'm running Emacs 27.1 + elpy 1.35.0 + python 3.9.1 on macOS 11.1. I assume I'll need put an import line at the beginning of rpc.py like from pathlib import Path Any suggestions?

[akshaybadola]

@xdxu I forgot to add from pathlib import Path. Which jedi version are you running? Have you tried with jedi==0.18.0 in elpy's virtualenv? or jedi==0.18.0 in your available (either system or user path)?

[xdxu]

@xdxu I forgot to add from pathlib import Path. Which jedi version are you running? Have you tried with jedi==0.18.0 in elpy's virtualenv? or jedi==0.18.0 in your available (either system or user path)?

I am using jedi==0.18.0 in my system (it is the only version available).

[akshaybadola]

@xdxu I forgot to add from pathlib import Path. Which jedi version are you running? Have you tried with jedi==0.18.0 in elpy's virtualenv? or jedi==0.18.0 in your available (either system or user path)?

I am using jedi==0.18.0 in my system (it is the only version available).

@xdxu can you let me know if the patch with config mentioned in 1877# works for you? Otherwise it might be some other issue you're facing. and some more debugging may be required.

[xdxu]

@akshaybadola I believe I made a change to elpy.el with the following

(defun elpy-multiedit-stop ()
  "Stop editing multiple places at once."
  (interactive)
  (dolist (ov elpy-multiedit-overlays)
    (delete-overlay ov))
  (setq elpy-multiedit-overlays nil)
  (if (get-buffer "*Elpy Edit Usages*")
      (kill-buffer "*Elpy Edit Usages*")))

and it didn't work.

And I thought the variable 'elpy-rpc-backend' no longer exists?

[akshaybadola]

@xdxu elpy-multiedit-stop is for elpy-multiedit-python-symbol-at-point. I'm not sure if elpy-rpc-backend existed or not TBH. I've had that in my python-stuff.el forever. On checking now, yeah it's dead.

It's usually better IMO that if you make the change after loading elpy and override the symbol in a separate file (usually some file loaded from your .emacs) and let the original source be.

I'm afraid in your case much more debugging would be needed if that didn't work. I don't think I've kept a record of the patches which I'd made to debug (mostly hacky print statements and lisp expressions). To do it properly I think some additional tests are needed and I'm a bit hard up on time right now.

[xdxu]

Tried a few more times and with no luck. It is weird it is working on my Linux (emacs 27.1 + python 3.9.1 + elpy 1.35.0 + jedi 0.18.0), without any patch.

[akshaybadola]

Hi @xdxu, that is indeed strange though I'm sure it's fixable. I'm not sure what the status is on the development of elpy right now. I've seen one fork which is tryting to incorporate pydantic (@gfederix https://github.com/gfederix/elpy perhaps he can chip in with his comments), which I think is a good idea but I don't know who's working on which feature/bug fix along which path. @galaunay would you please tell us a bit about what's going on with elpy, the development and the status? I've used it for a long time and would love to help out.

[gfederix]

Hi @akshaybadola @xdxu, I have fixed jedi 0.18 compatibility issues in my repository and thinking about improving the code. I would be grateful if my code gets into the main repository. The code passes all tests as well as work in emacs. But I must warn you that I dropped support for Python 2.7 in JSON-RPC server code. I am also still thinking about a better architecture for pydantic integration. You can use my code using stright package for example: (use-package elpy :straight (elpy :fork (:host github :repo "gfederix/elpy" :branch "dev")))

[xdxu]

@gfederix I can confirm the latest dev branch in your fork is working. Thanks for the efforts! I'm now hoping that your changes will be taken by the upstream soon.

[sten0]

@gfederix, thank you for your work on this, and thank you for dropping py2 support; it's been EOL for over a year now, after all! Also, thank you for dropping jedi16 compat; there were enough 17-to-18 issues that this seems reasonable, not to mention a wise use of time. Would you please clean up your dev branch's history and submit a PR?

[sten0]

P.S. and please "@sten0" so I don't miss it (I'm the maintainer of the Debian package)

[gfederix]

Hi @sten0. I was going to do some more code this weekend and probably do a pull request next weekend (February 6th). I also found that lisp code doesn't use requirements-rpc3.6.txt and requirements-rpc.txt, for which they were obviously created. That in turn caused the current problems. It's strange that @galaunay didn't update the lisp code after creating the files. I'm considering adding fixes. But for this, the files need to be added in MELPA reсipes.

[JulienPalard]

Having the issue today, I tried with Python 3.8.7 and have the same issue:

Elpy Configuration

Emacs.............: 27.1
Elpy..............: 1.35.0
Virtualenv........: .venv (/home/mdk/clones/hkis/hkis-exercises/exercises/.venv/)
Interactive Python: python 3.8.7 (/home/mdk/clones/hkis/hkis-exercises/.venv/bin/python)
RPC virtualenv....: rpc-venv (/home/mdk/.emacs.d/elpy/rpc-venv)
 Python...........: python 3.8.7 (/home/mdk/.emacs.d/elpy/rpc-venv/bin/python)
 Jedi.............: 0.18.0
 Rope.............: Not found (0.18.0 available)
 Autopep8.........: Not found (1.5.4 available)
 Yapf.............: Not found (0.30.0 available)
 Black............: 20.8b1
Syntax checker....: flake8 (/home/mdk/.local/.venv/bin/flake8)

giving:

Traceback (most recent call last):
  File "/home/mdk/.emacs.d/elpa/elpy-20201115.1811/elpy/rpc.py", line 104, in handle_request
    self.write_json(result=result,
  File "/home/mdk/.emacs.d/elpa/elpy-20201115.1811/elpy/rpc.py", line 77, in write_json
    self.stdout.write(json.dumps(kwargs) + "\n")
  File "/home/mdk/.local/lib/python3.8/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/home/mdk/.local/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/home/mdk/.local/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/home/mdk/.local/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type PosixPath is not JSON serializable

[sten0]

Hi @gfederix

Hi @sten0. I was going to do some more code this weekend and probably do a pull request next weekend (February 6th). I also found that lisp code doesn't use requirements-rpc3.6.txt and requirements-rpc.txt, for which they were obviously created. That in turn caused the current problems. It's strange that @galaunay didn't update the lisp code after creating the files. I'm considering adding fixes. But for this, the files need to be added in MELPA reсipes.

I'm not sure that any lisp code is supposed to use requirements-rpc*.txt; as far as I tell these files are exclusively consumed by pip via scripts/setup. What do you have in mind (other than hopefully to set the Jedi requirement to 0.18.0)?

Re: your future PR, thank you, I truly appreciate it! Elpy is scheduled to be dropped from Debian 08 Feb, most likely at midnight UTC, and at the moment there's nothing I can do other than have faith that your work will solve the release critical issues that are precipitating that action.

Regards, Nicholas

[jacknashg]

hit the same issue as originally opened by @akshaybadola , but solved it in a different way.

my fix is to modify jedibackend.py, change the last line of function rpc_get_definition like this:

def rpc_get_definition(self, filename, source, offset):
    line, column = pos_to_linecol(source, offset)
    # ...
    # return (loc.module_path, offset)
    return [str(loc.module_path), offset]

it seems, 2 factors together result in the issue 1, if a tuple is returned here, it will (strangely) be lost, and the calling function rpc.py:handle_request won't get anything. returning a list is ok

2, "loc.module_path" is of type "PosixPath", and the calling function (elpy-rpc.el:elpy-rpc--call-blocking) cannot handle it right, but a plain str is ok

[rksm]

Any reason why the fix isn't integrated? Currently goto-definition is still broken on MacOS, python 3.7, 3.8, 3.9. I can confirm that the fix here successfully resolves it.