elnode curl: (56) Malformed encoding found in chunked-encoding

Some .org files fail to render in any browser when served by elnode. These files export without error, and the rendered html can be validated using http://validator.w3.org/. Only curl gives a clue about malformed encoding. It might just be that elnode is doing such a good job of being asynchronous that it tries to write data before it is available.

Tested on Emacs 24.4.1 with Org-mode 8.2.10 and elnode-0.9.9.8.8, also on Emacs 24.5.1 with Org-mode 8.2.10 and elnode-0.9.9.8.8 (same behaviour for both).

How to reproduce (from emacs -Q):

Install elnode somewhere. I temporarily moved my emacs.d and installed from marmalade.
Evaluate this snippet. ~/dotemacs/emacs.org is cloned from https://github.com/bzg/dotemacs (arbitrary large and public org file).

(require 'elnode)

(defun org2html (src)
  (let* ((org-inhibit-startup t)
         (visitingp (find-buffer-visiting src))
         (work-buffer (or visitingp (find-file-noselect src)))
         (html (with-current-buffer work-buffer
                 (org-export-as 'html nil t nil
                                '(:html-doctype "html5" :html-html5-fancy t)))))
    html))

(defun my-test-handler (httpcon)
  "Demonstration function"
  (let ((html (org2html "~/dotemacs/emacs.org")))
    (elnode-send-html httpcon html)))

(elnode-start 'my-test-handler :port 9999 :host "localhost")

Try to access in the terminal emulator:

~ $ curl localhost:9999
<!DOCTYPE html>
<html>
<head>
<title>bzg .emacs.el file</title>
--- SNIP ---
<p class="date">Created: 2015-07-13 Mon 11:42</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.4.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
curl: (56) Malformed encoding found in chunked-encoding
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate<~ $

You will see something like the following in *elnode-server-error* buffer:

2015-07-13T11:34:35: elnode-http-start: starting HTTP response on *elnode-webserver-localhost:9999* <127.0.0.1:42417>                                                                                          
2015-07-13T11:34:35: elnode-http-send-string *elnode-webserver-localhost:9999* <127.0.0.1:42417> [[<!DOCTY...]]                                                                                                
2015-07-13T11:34:35: elnode-http-send-string *elnode-webserver-localhost:9999* <127.0.0.1:42417> [[]]   
2015-07-13T11:34:35: elnode--http-end ending socket *elnode-webserver-localhost:9999* <127.0.0.1:42417>

I see the same behaviour using this, and some of my own files that I don't want to share just yet.

Expected behaviour: same for that of smaller org files. Here's an example ~/test.org that I used:

#+title: test.org                                                                                       
#+author: Ruben Maher                                                                                   
#+date: <2015-07-13 Mon 09:25:26>                                                                       

* This is a test to show off =elnode=.

  It's a pretty cool webserver.

Re-evaluate my-test-handler, changing the argument to org2html:

(defun my-test-handler (httpcon)
  "Demonstration function"
  (let ((html (org2html "~/test.org")))
    (elnode-send-html httpcon html)))

Try to access in a terminal emulator:

~ $ curl localhost:9999
<!DOCTYPE html>
<html>
<head>
<title>test.org</title>
--- SNIP ---
<p class="date">Created: 2015-07-13 Mon 11:48</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.4.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>~ $

No problem. It renders correctly in a web browser. *elnode-server-error* now shows:

2015-07-13T11:48:59: elnode-http-start: starting HTTP response on *elnode-webserver-localhost:9999* <127.0.0.1:44466>
2015-07-13T11:48:59: elnode-http-send-string *elnode-webserver-localhost:9999* <127.0.0.1:44466> [[<!DOCTY...]]
2015-07-13T11:48:59: elnode-http-send-string *elnode-webserver-localhost:9999* <127.0.0.1:44466> [[]]
2015-07-13T11:48:59: elnode--http-end ending socket *elnode-webserver-localhost:9999* <127.0.0.1:44466>

Jul 13 '15 02:07 eqyiel

I found the problem. It's a one-line fix so I'll just post it here.

In elnode-http-send-string, (format %x (length str)) doesn't always equal (format %x (string-bytes str). elnode tells the client to expect (string-bytes str), but the chunk size in the header is determined by (length str)!

I guess this is a bug in Emacs C source code. (Edit: this behaviour is actually documented. See here and here, "if the string contains multibyte characters, this is not necessarily the number of bytes in the string; it is the number of characters.").

Demonstration, using this file

(defun org2html (src)
  (let* ((org-inhibit-startup t)
         (visitingp (find-buffer-visiting src))
         (work-buffer (or visitingp (find-file-noselect src)))
         (html (with-current-buffer work-buffer
                 (set-buffer-file-coding-system 'utf-8)
                 (org-export-as 'html nil t nil
                                '(:html-doctype "html5" :html-html5-fancy t))
                 )))
    html))

(let ((thing (org2html "~/dev/emacs-notes/videos.org")))
  (message (concat "(string-bytes thing) is %x, (length thing) is %x. "
                   "That's %d bytes shorter than it should be!")
           (string-bytes thing) (length thing)
           (- (string-bytes thing) (length thing))))

;; => "(string-bytes thing) is 5dc5, (length thing) is 5dc3. That's 2 bytes
;; shorter than it should be!"

TL;DR: (format "%x\r\n%s\r\n" (length str) (or str ""))) should be (format "%x\r\n%s\r\n" len (or str ""))) (len already calculated in let)

Jul 15 '15 00:07 eqyiel

Unicode characters will cause this problem.

On Tue, Jul 14, 2015 at 8:44 PM, Ruben Maher [email protected] wrote:

I found the problem. It's a one-line fix so I'll just post it here.

In elnode-http-send-string https://github.com/nicferrier/elnode/blob/master/elnode.el#L1783, (format %x (length str)) doesn't always equal (format %x (string-bytes str). elnode tells the client to expect (string-bytes str), but the chunk size in the header is determined by (length str)!

I guess this is a bug in Emacs C source code.

Demonstration, using this file https://github.com/sachac/emacs-notes/blob/gh-pages/videos.org

(defun org2html (src) (let* ((org-inhibit-startup t) (visitingp (find-buffer-visiting src)) (work-buffer (or visitingp (find-file-noselect src))) (html (with-current-buffer work-buffer (set-buffer-file-coding-system 'utf-8) (org-export-as 'html nil t nil '(:html-doctype "html5" :html-html5-fancy t)) ))) html))

(let ((thing (org2html "~/dev/emacs-notes/videos.org"))) (message (concat "(string-bytes thing) is %x, (length thing) is %x. " "That's %d bytes shorter than it should be!") (string-bytes thing) (length thing) (- (string-bytes thing) (length thing))))

;; => "(string-bytes thing) is 5dc5, (length thing) is 5dc3. That's 2 bytes ;; shorter than it should be!"

TL;DR: (format "%x\r\n%s\r\n" (length str) (or str ""))) should be (format "%x\r\n%s\r\n" len (or str ""))) (len already calculated in let)

— Reply to this email directly or view it on GitHub https://github.com/nicferrier/elnode/issues/98#issuecomment-121436966.

Jul 15 '15 01:07 joelmccracken

Right. Not sure what elnode should do. Automatically convert? (how would we send bytes?) or make it clear that it's not really a string... it's a string of bytes?

Jul 15 '15 10:07 nicferrier

Unless I've completely misunderstood something (wouldn't be the first time), there's no conversion necessary? The problem is just that size of the chunk reported in the header (%x\r\n) is not always correct when it is retrieved with length (if the string contained unicode characters for instance). The chunk still contains a string and the function still sends a string, but elnode should figure out the chunk size with string-bytes instead of length.

Jul 15 '15 14:07 eqyiel

I think that's option 2, accept that send-string is sending bytes, not strings.

Either way it needs to change, but it could be either way.

Jul 15 '15 14:07 nicferrier

I think it should send the number of bytes.

On Wed, Jul 15, 2015 at 10:29 AM, Nic Ferrier [email protected] wrote:

I think that's option 2, accept that send-string is sending bytes, not strings.

Either way it needs to change, but it could be either way.

— Reply to this email directly or view it on GitHub https://github.com/nicferrier/elnode/issues/98#issuecomment-121633832.

Jul 15 '15 14:07 joelmccracken

What's the status on this? I just ran into this issue in 2017.

Feb 11 '17 15:02 jiri

@jiri It's been a long time, but IIRC you need to monkeypatch elnode-http-send-string changing (length str) to (string-bytes str) on this line: https://github.com/nicferrier/elnode/blob/master/elnode.el#L1783

Edit: or you can just use the len which is let-bound, see TLDR here: https://github.com/nicferrier/elnode/issues/98#issuecomment-121436966

Feb 12 '17 22:02 eqyiel

Thanks!

I've been looking into making a patch for this, but there already is a pull request. It would be nice if @nicferrier either accepted it or appointed a maintainer of the package. I think elnode is useful enough that it should be maintained :)

That being said, the issue seems to have disappeared as I have not been able to replicate it since I restarted Emacs. If it ever surfaces, I'll bump this issue again :)

Feb 15 '17 18:02 jiri