Emacs process sentinel/plz.el issue ("wrong-type-argument plz-response nil")

[fleimgruber]

Emacs 27.2.50 OS Windows 10 ement 20210928.943

On ement-connect I get the backtrace:

Debugger entered--Lisp error: (wrong-type-argument plz-response nil)
  signal(wrong-type-argument (plz-response nil))
  ement--hostname-uri("matrix.org")
  #f(compiled-function () #<bytecode 0x89ff194f11>)()
  ement-connect(:user-id "@redacted:matrix.org" :password "redacted")
  funcall-interactively(ement-connect :user-id "@redacted:matrix.org" :password "redacted")
  call-interactively(ement-connect record nil)
  command-execute(ement-connect record)
  helm-M-x-execute-command(ement-connect)
  helm-execute-selection-action-1()
  helm-execute-selection-action()

any ideas on further debugging this? Minimal reproducer is (ement--hostname-uri "matrix.org") in which eventually (plz 'get "https://matrix.org/.well-known/matrix/client" :as 'response :then 'sync) is nil.

[alphapapa]

Thanks, this is an issue with Emacs's process sentinels and plz that I've been trying to solve, but I haven't been able to make much progress on. See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=50166.

The next time this happens, please check M-x list-processes and see how many child processes Emacs has; I don't know if that's related, but it might be.

FWIW, only one or two users have reported this issue, and they've seemed to say that, if they retried a few times, it would eventually work, and that once it did, it would work normally.

If I can't solve this soon, I might have to add a workaround to use url-retrieve-synchronously just for ement--hostname-uri, which might help it get past this issue so the rest of the client can work normally.

[fleimgruber]

Thanks for the heads-up. I tried a few times manually, but could not get it working. I also did M-x list-processes, but I only see the expected 2 (Emacs server and emacsql-sqlite for org-roam). Maybe you have additional ideas to debug this? I can also imagine this having something to do with the way processes work on Windows, would not be the first time. What does plz have as requirements and expectations for the execution environment?

[alphapapa]

Thanks. There are no expectations other than having curl installed. Emacs should abstract the rest (of course, there might be some platform-specific bugs at play, but I don't think this issue is exclusive to Windows, AFAIK).

The only thing I know of to debug it further is to keep trying to isolate the problem described in the linked Emacs bug report, trying to produce a minimal test case that might aid the Emacs developers in fixing any bug that might exist in process.c. That's a long-term goal of mine, but having very little knowledge of Emacs's process-related internals, my progress on it is very slow, and I'm not working on it at the moment.

One of the things that confounds me is that loading room avatars seems to work without any problem, and that involves making many plz requests in rapid succession, which are run in parallel--that would seem to suggest that the number of outstanding requests or processes doesn't matter. But in the ECM in the bug report, the number of processes does matter; and when running the plz tests in ERT, that also seems to matter. So it's hard to understand where exactly the problem lies, and it's not easy to reproduce it in a meaningful way, it seems.

[Oderjunkie]

i'm able to reproduce the problem consistently in an arch linux VM [running on windows, with internet connection], using GNU emacs 27.2, and using ement.el 20210924.1051. M-x list-processes shows 0 procs before executing ement-connect, after executing it, list-processes shows 2 procs, both of which are plz. the first proc has no PID, no command, and a status of open, but the second has a PID that changes, a command of curl --silent --compressed --location --dump-header - --config -, and a status of run.

[alphapapa]

@Oderjunkie Thanks. What version of curl is installed? What happens if you restart Emacs a few times, does it ever work? When this problem happens to me, which is probably less than 5% of the time I ement-connect, it usually works if I try again, and then it continues working for days without any problems.

[Oderjunkie]

@alphapapa curl version 7.77.0, and in my several attempts over 2 days with multiple emacs restarts, it happens 100% of the time for me =/

[alphapapa]

Thanks. I'm afraid I don't know what else to suggest. Since it works for me, using both older and newer Emacs versions, and also on Linux, I don't know what the problem could be. If you can, try it on a "bare metal" Linux system rather than in a VM, and maybe on a different Linux distro, but that shouldn't make any difference.

[Oderjunkie]

update: i never thought about it, but when emacs is executed with a GUI [i'm using Xorg], it... works fine? that's weird.

[alphapapa]

@Oderjunkie So it always, and only, fails when you run emacs -nw?

[Oderjunkie]

@alphapapa yup. doesnt matter if i do emacs -nw in the builtin linux tty, or in alacritty, it always fails. but as soon as i get rid of -nw, boom, works. weird.

[alphapapa]

@Oderjunkie Well, a concrete data point like that might eventually point to the problem, so thanks.

[Robert-Forrest]

Commenting in case this is also a useful data point - I have been 100% unable to connect to my matrix server using M-x ement-connect with @user:server followed by password. However, manually calling M-: (ement-connect :user-id @user:server :password *** :uri-prefix server) works for me.

For examples of user-id and uri-prefix formatting, I have something like:

:user-id @me:matrix.myserver.com :uri-prefix https://matrix.myserver.com

Thanks for this package!

[binarydigitz01]

Can 1+ on @Robert-Forrest, Doing by his way, i'm able to chat on matrix. Using arch linux with emacs 28.

[alphapapa]

@Robert-Forrest @Ice-Cube69 Are your servers properly configured with the /.well-known URI so the Matrix server URI prefix can be automatically discovered?

[Robert-Forrest]

No, my matrix server was returning a 404 on /.well-known/matrix/server, adding: serve_server_wellknown: true to my homeserver.yaml such that Synapse serves the m.server JSON data properly seems to have solved this issue!

Thanks!

[altomcat]

Hello,

On windows 10 I solved this issue by switching from this (default) curl release

λ \Windows\System32\curl.exe -V
curl 7.83.1 (Windows) libcurl/7.83.1 Schannel
Release-Date: 2022-05-13
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS HSTS IPv6 Kerberos Largefile NTLM SPNEGO SSL SSPI UnixSockets

with the last available from https://curl.se/windows/

λ curl -V
curl 8.0.1 (x86_64-w64-mingw32) libcurl/8.0.1 OpenSSL/3.1.0 (Schannel) zlib/1.2.13 brotli/1.0.9 zstd/1.5.5 WinIDN libssh2/1.10.0 nghttp2/1.52.0 ngtcp2/0.14.1 nghttp3/0.10.0 libgsasl/2.2.0
Release-Date: 2023-03-20
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli gsasl HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI threadsafe TLS-SRP UnixSockets zstd

Then I changed the path as a priority to the last release and make some basic checking with (executable-find "curl.exe") to find the default application used by plz.

Finaly I checked that everything was fine with the following elisp code.

(require 'plz)
(plz 'get "https://httpbin.org/user-agent")

Thanks for your work!

[alphapapa]

Hi Arnaud,

Thanks for reporting that; it could be a very useful data point.

Since this is your first message on this issue, would you please explain more about the problem you experienced? How often did it happen? What version of Emacs and plz are you using? What build of Emacs are you using? How long did you notice the problem happening before you experimented with other versions of curl to try to solve it? Do you only run Windows, or did you also notice any similar problems on other platforms?

Thanks for your help.

[altomcat]

This was my first time using matrix and I wanted to test a client for Emacs. Thanks to System Crafters to point me out. Every time I wanted to connect, it failed with the following error just after typing my ID and my password:

"c:/Windows/System32/curl.exe"
error in process sentinel: plz: plz: Curl error: "plz--sentinel: Curl error", #s(plz-error (2 . "Failed to initialize.") nil nil)
error in process sentinel: plz: Curl error: "plz--sentinel: Curl error", #s(plz-error (2 . "Failed to initialize.") nil nil)

Debugger entered--Lisp error: (ement-api-error "2: Failed to initialize.")
  signal(ement-api-error ("2: Failed to initialize."))
  ement-api-error(#s(plz-error :curl-error (2 . "Failed to initialize.") :response nil :message nil))
  plz--sentinel(#<process plz-request-curl> "exited abnormally with code 2\n")

I'm currently running GNU Emacs 28.2 (build 2, x86_64-w64-mingw32) of 2022-09-13. plz is 0.5.4

I didn't kwow about the variable plz-curl-program until now. It's a more convenient way to go for me. I haven't tried other distros yet but it migth happen soon.

[alphapapa]

@altomcat I see, that's actually a different problem: the "Failed to initialize" error means that curl didn't recognize some of the arguments it was given, likely because the one that came with Windows is incompatible with actual curl. So, indeed, installing "real" curl is the solution to that.

Thanks for letting me know about that. Since this issue is about a different problem, I may hide these messages, but they're a good future reference.

[kaibagley]

Sorry to add more to this, but @altomcat, how did you change the default curl to be that of curl.se's curl? Adding it to PATH doesn't seem to stop powershell from using its default, even if the file is at the from. And how would you use plz-curl-program to do this? I've tried setting the variable to the path to the curl.se curl.exe, which causes everything to return nil.

[alphapapa]

@kaibagley Please open a new issue to discuss that problem. This issue is about a particularly difficult, longstanding problem that's already hard enough to understand.

[alphapapa]

AFAIK this is solved in plz v0.7.