Proxy server

[alanthird]

Always the bane of any http based project, I'm behind a corporate firewall and can only access the internet through a proxy server.

I already set the proxy server in Emacs using url-proxy-services, so it might be nice to automatically pick up proxy server settings from it.

I tried working round it myself by adding a "--proxy" flag to plz-curl-default-args, but it fails as curl returns a second set of headers for the proxy server:

> curl.exe --silent --location --dump-header - --proxy http://localhost:3128 "https://httpbin.org/user-agent"
HTTP/1.1 200 Connection established
Server: SimpleHTTP/0.6 Python/3.7.3
Date: Wed, 25 Aug 2021 12:18:20 GMT
Proxy-Agent: SimpleHTTP/0.6 Python/3.7.3

HTTP/1.1 200 OK
Date: Wed, 25 Aug 2021 12:18:20 GMT
Content-Type: application/json
Content-Length: 34
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

{
  "user-agent": "curl/7.55.1"
}

[alphapapa]

Hi Alan,

Thanks, this is just the kind of thing I need testers to let me know about. Picking up proxy settings from that variable might be good, but since it's part of the url library, I'm not sure. On the other hand, it is built-in to Emacs, so...why not?

Anyway, I'll have to modify the response-parsing function to handle this. I probably won't get to this very soon, as I've got some other things on my plate. But thanks for letting me know. :)

[alphapapa]

Hi Alan,

I added support for Curl's proxy header output, so the example you gave should work now. Please let me know if it does for you.

Having solved that, what do you think we should do regarding setting proxy settings for plz? i.e. should we add options for that, or use url's options, or use environment variables, or...?

Thanks for your help and your patience.

[sawyerzheng]

@alphapapa I'm a network novice. Don't know if this is still a problem.

With latest commit:

1. without proxy:

(plz 'get "https://httpbin.org/user-agent")
"{
  \"user-agent\": \"curl/7.86.0\"
}
"

2. with proxy enabled:

(plz 'get "https://httpbin.org/user-agent")
"HTTP/2 200 
date: Fri, 09 Dec 2022 06:18:49 GMT
content-type: application/json
content-length: 34
server: gunicorn/19.9.0
access-control-allow-origin: *
access-control-allow-credentials: true

{
  \"user-agent\": \"curl/7.86.0\"
}
"

[alphapapa]

@sawyerzheng

1. How are you configuring your system to use the proxy?
2. What happens if you add the curl argument --http1.1 to disable HTTP/2 support?

Thanks for your help.

[sawyerzheng]

--http1.1 works. Not only on system proxy, but also on proxy inside emacs.

;; no --http1.1 and no proxy
(setq plz-curl-default-args '("--silent" "--compressed" "--location" "--dump-header" "-"))
(plz 'get "https://httpbin.org/user-agent")
;; promt errror:
Debugger entered--Lisp error: (plz-http-error "plz--response: Unable to parse HTTP response" "HTTP/1.1 200 Connection established\15\n\15\nHTTP/2 200 ...")
plz--response()
plz--sentinel(#<process plz-request-curl> "finished\n")
plz(get "https://httpbin.org/user-agent")
(progn (plz 'get "https://httpbin.org/user-agent"))
elisp--eval-last-sexp(t)
eval-last-sexp(t)
eval-print-last-sexp(nil)
funcall-interactively(eval-print-last-sexp nil)
command-execute(eval-print-last-sexp)




;; with --http1.1 added and disabled proxy
(cl-pushnew "--http1.1" plz-curl-default-args :test #'equal)
("--http1.1" "--silent" "--compressed" "--location" "--dump-header" "-")
(plz 'get "https://httpbin.org/user-agent")
"{
  \"user-agent\": \"curl/7.86.0\"
}
"

;; with --http1.1 and enable proxy just within emacs with setenv
(plz 'get "https://httpbin.org/user-agent")
"{
  \"user-agent\": \"curl/7.86.0\"
}
"

;; with --http1.1 and enable system proxy
(plz 'get "https://httpbin.org/user-agent")
"{
  \"user-agent\": \"curl/7.86.0\"
}
"

And it need to correct that my last post was wrong. After a manually delete old plz folder and plt-<...>-.eln files. I cannot repeat this result mensioned in the last post. Sorry for it.

2. with proxy enabled:

(plz 'get "https://httpbin.org/user-agent")
"HTTP/2 200 
date: Fri, 09 Dec 2022 06:18:49 GMT
content-type: application/json
content-length: 34
server: gunicorn/19.9.0
access-control-allow-origin: *
access-control-allow-credentials: true

{
  \"user-agent\": \"curl/7.86.0\"
}
"

[alphapapa]

@sawyerzheng Ok, let's try to fix this remaining issue:

;; no --http1.1 and no proxy
(setq plz-curl-default-args '("--silent" "--compressed" "--location" "--dump-header" "-"))
(plz 'get "https://httpbin.org/user-agent")
;; promt errror:
Debugger entered--Lisp error: (plz-http-error "plz--response: Unable to parse HTTP response" "HTTP/1.1 200 Connection established\15\n\15\nHTTP/2 200 ...")

I can't see in that output why it would be failing to parse that response, but I just pushed another commit which should improve that error message. So please install the latest commit (and you might find this helpful for reloading a new version of a package), and then evaluate this:

(let ((eval-expression-print-level nil)
      (eval-expression-print-length nil)
      (plz-curl-default-args '("--silent" "--compressed" "--location" "--dump-header" "-")))
  (plz 'get "https://httpbin.org/user-agent"))

Thanks.

[sawyerzheng]

;; disable  proxy ----------------------
❯ curl --silent --location --dump-header -  "https://httpbin.org/user-agent"
HTTP/2 200
date: Mon, 12 Dec 2022 05:44:25 GMT
content-type: application/json
content-length: 34
server: gunicorn/19.9.0
access-control-allow-origin: *
access-control-allow-credentials: true

{
  "user-agent": "curl/7.86.0"
}

;; elisp
(let ((eval-expression-print-level nil)
      (eval-expression-print-length nil)
      (plz-curl-default-args '("--silent" "--compressed" "--location" "--dump-header" "-")))
  (plz 'get "https://httpbin.org/user-agent"))
;; debug result
Debugger entered--Lisp error: (plz-http-error "plz--response: Unable to parse HTTP response statu..." "HTTP/2 200 \15")
  plz--response()
  plz--sentinel(#<process plz-request-curl> "finished\n")
  plz(get "https://httpbin.org/user-agent")
  (let ((eval-expression-print-level nil) (eval-expression-print-length nil) (plz-curl-default-args '("--silent" "--compressed" "--location" "--dump-header" "-"))) (plz 'get "https://httpbin.org/user-agent"))
  (progn (let ((eval-expression-print-level nil) (eval-expression-print-length nil) (plz-curl-default-args '("--silent" "--compressed" "--location" "--dump-header" "-"))) (plz 'get "https://httpbin.org/user-agent")))
  elisp--eval-last-sexp(t)
  eval-last-sexp(t)
  eval-print-last-sexp(nil)
  funcall-interactively(eval-print-last-sexp nil)
  command-execute(eval-print-last-sexp)


;; enabled  proxy -----------------------------------------

❯ curl --silent --location --dump-header - --proxy http://172.30.48.1:7890 "https://httpbin.org/user-agent"
HTTP/1.1 200 Connection established

HTTP/2 200
date: Mon, 12 Dec 2022 05:43:03 GMT
content-type: application/json
content-length: 34
server: gunicorn/19.9.0
access-control-allow-origin: *
access-control-allow-credentials: true

{
  "user-agent": "curl/7.86.0"
}


;; elisp 
(let ((eval-expression-print-level nil)
      (eval-expression-print-length nil)
      (plz-curl-default-args '("--silent" "--compressed" "--location" "--dump-header" "-")))
  (plz 'get "https://httpbin.org/user-agent"))

;; debug result
Debugger entered--Lisp error: (plz-http-error "plz--response: Unable to parse HTTP response statu..." "HTTP/2 200 \15")
  plz--response()
  plz--sentinel(#<process plz-request-curl> "finished\n")
  plz(get "https://httpbin.org/user-agent")
  (let ((eval-expression-print-level nil) (eval-expression-print-length nil) (plz-curl-default-args '("--silent" "--compressed" "--location" "--dump-header" "-"))) (plz 'get "https://httpbin.org/user-agent"))
  (progn (let ((eval-expression-print-level nil) (eval-expression-print-length nil) (plz-curl-default-args '("--silent" "--compressed" "--location" "--dump-header" "-"))) (plz 'get "https://httpbin.org/user-agent")))
  elisp--eval-last-sexp(t)
  eval-last-sexp(t)
  eval-print-last-sexp(nil)
  funcall-interactively(eval-print-last-sexp nil)
  command-execute(eval-print-last-sexp)

[alanthird]

I'm not sure if I'm doing something wrong, but this is returning nil:

(let ((plz-curl-default-args '("--silent" "--location" "--dump-header"
                                 "-" "--proxy" "http://localhost:3128")))
  (plz 'get "https://httpbin.org/user-agent"))

I have a nasty feeling it may be because I'm now apparently using the built-in Windows curl, which is a recent addition, and perhaps there are line-ending issues or something. If you think the above should work, I'll investigate further.

(I already had to remove --compressed because the Windows version doesn't support it. 🙄)

[alphapapa]

This should now be fixed in v0.3, with both HTTP/1.1 and HTTP/2. Please let me know how it works for you. Thanks to both of your for your help.