Cannot retrieve list of official docsets

[dchrzanowski]

I've narrowed down that helm-install-docset executes helm-dash-read-json-from-url that seems to fail to retrieve data from "https://api.github.com/repos/Kapeli/feeds/contents/" with the error: Wrong type argument: integer-or-marker-p, nil. Possibly a change in the way url-http-end-of-headers is set/not-set?

[kidd]

This looks like a dup of #132. It is a connectivity issue trying to fetch that json.

I can't reproduce it in my machine, and even we could add some protection so it doesn't crash hard (PRs accepted), I'm not sure it'd solve the real issue.

[dchrzanowski]

I'll try again today, see if there is problem in reproducing the error.

[buhudu]

Same problem here! Any suggestions?

[kidd]

depending on your elisp debugging skills you can try to call edebug-defun on the breaking function and try to see why the variable url-http-end-of-headers is nil.

There's this reported bug https://emacs.stackexchange.com/questions/32950/why-isnt-url-http-end-of-headers-set-correctly-when-using-url-automatic-caching, but the behavior is different, so it being nil probably means we got no content at all from the request. But there's no reason why you're getting failures from downloading from that url.

[dchrzanowski]

Yes I've already looked at this post from stack before. There really is no reason to not get any data back since the url is live.

[buhudu]

Weird, when I step through the debugger on url-retrieve in url.el I can get to the point where I select a docset to install. But then I get a "Bad URL" error.

Outside of the debugger I don't even make it to that step.

[dchrzanowski]

That's some odd behaviour. My Elisp fu is not good enough to tackle this.

[buhudu]

It works. Thank you.

[dchrzanowski]

Alternative:

You can also go to the list of feeds here. Click the desired xml file. Then download the tgz file from any of the url's from the xml file. Once you have the file downloaded, you can run M-x helm-dash-install-docset-from-file and point it to the downloaded file.

[kidd]

Here's another alternative. I rewrote the function so that it uses curl but returns the same result. Works on my machine (TM)

(defun helm-dash-read-json-from-url (url)
  (shell-command (concat "curl -s " url) "*helm-dash-download*")
  (with-current-buffer "*helm-dash-download*"
    (json-read)))

[jeeger]

I've replaced all the URL functions with calls to curl, since it seems curl's proxy handling is a bit better than url.el's. The change is available here in the windows-fixes branch (should also work on Linux☺).

[jbaum98]

I think that this a problem with url-retrieve-synchronously. This works:

(url-retrieve "https://api.github.com/repos/Kapeli/feeds/contents"
  (lambda (status) (switch-to-buffer (current-buffer)))))

As a side note, if we do want to switch to curl I think we should use request.el

[jeeger]

I think that this a problem with url-retrieve-synchronously. This works:

(url-retrieve "https://api.github.com/repos/Kapeli/feeds/contents"
  (lambda (status) (switch-to-buffer (current-buffer)))))

Nope, this fails here as well.

As a side note, if we do want to switch to curl I think we should use request.el

Yes, that would make much more sense. I'm not familiar with request.el, but I'll have a look at it if my ad-hoc curl patching fails ☺.

[jbaum98]

Nope, this fails here as well.

What do you mean by "here"? I just meant that evaluating that using eval-expression gives me a buffer with a response in it, whereas

(switch-to-buffer (url-retrieve-synchronously "https://api.github.com/repos/Kapeli/feeds/contents"))

doesn't work.

[jeeger]

I mean that evaluating this expression does not return any content on this PC, since url.el is broken for using HTTPS over a proxy. This is why I've worked around url.el by using curl.

[jbaum98]

Maybe I've misunderstood, but this makes me think that the proxy is not the issue. Because url-retrieve works for me even though url-retrieve-synchronously doesn't. So at least there are 2 issues: the proxy messes up everything, but even without the proxy I'm experiencing this issue with helm-dash.

[jeeger]

Interesting, here it's definitely the proxy. request.el definitely looks like a good alternative here, maybe gated with an availability check so it's only used when installed.

[kidd]

@jeeger , I'd buy into it if it's gated. I don't want to make an extra dependency for everyone.

Total naive question. Is a fix for proxy over http possible by plumbing some elisp? or we'd need hardcore networking knowledge for that? upstream emacs would benefit from this fix if we'd have it.

[jeeger]

I've had a look at the url.el elisp, and it's a tangled mess for me. I think that bug that might well have it's origin in the basic structure of url.el. TLS and Proxies are both kinds of "stream processors" that might not be able to be "stacked" recursively. I don't see me fixing url.el in the time I have available, even though it would of course be preferable to using an external package.

Using request.el would be much easier, and it could be easily gated behind a featurep invocation.

Edit: What makes this more complicated is that it seems this bug only occurs with redirects.