Static Files Truncated at 65,339 or 65340 via TLS connection

[daninus14]

In Chrome I am getting the following error for some Static Files from the server

net::ERR_CONTENT_LENGTH_MISMATCH 200 (OK)

The headers seem to be alright, however the actual files are being truncated

connection:    keep-alive
content-length:    162770
content-type:    text/css; charset=utf-8

The actual files being transferred are truncated partially and only have 65,339 or 65340 characters in length.

I just tried changing back to Hunchentoot with the caveman2 app and it works. Actually I just checked and interestingly enough the actual length of characters in the file is 162762 but the content-length header has 162770. It could be some escape characters that the browser just renders. Unclear. Nevertheless, this works with Hunchentoot and not with woo when removing :server :woo.

It looks like somewhere in woo it's limiting the file size to be sent over to 65,339 or 65,340. Just want to point out that 2^16 = 65536. So I imagine somewhere in the code there must be an integer handling how much to actually transfer, and because it's maxed out, it truncates the transfer. Probably changing this to a larger integer of say 32 bits is not such a good idea, because any file longer than that (I think about 4 GB) will then just be truncated as well. I wonder where the content-length header is being set, from reading the file size in the filesystem probably, and it could make sense to look into that code as a sample to dealing with the integers. Setting it to an integer 64 bits long will probably solve the issue for a long time though.

I'll be happy to help if you can point out to me which files and functions are dealing with the file transfer.

[fukamachi]

Thank you for reporting!

Is this only when SSL is enabled, or always? It may be because of a new function wev:write-socket-stream which sends a static file.

[daninus14]

Hi, finally got around to checking it. I can confirm this only happens when SSL is enabled.

I see here is the place where that is used: https://github.com/fukamachi/woo/blob/7f5219c55d49190f5ae17b123a8729b31c5d706e/src/woo.lisp#L391

And just below seems to be without SSL https://github.com/fukamachi/woo/blob/7f5219c55d49190f5ae17b123a8729b31c5d706e/src/woo.lisp#L405

It looks like the non SSL version is using this function: https://github.com/fukamachi/woo/blob/7f5219c55d49190f5ae17b123a8729b31c5d706e/src/ev/socket.lisp#L357

Not sure how that last function is actually sending the file, just by doing setf on sendfile-fd? Why can't we do the same by SSL? What about write-socket-stream is unique to SSL that it was needed to begin with?

[fukamachi]

Thank you for your efforts to look into it, but I still couldn't reproduce it. Woo can serve a 147KB PNG file and a 385 KB YAML file via HTTPS on my machine. It can be an environmental issue.

Not sure how that last function is actually sending the file, just by doing setf on sendfile-fd? Why can't we do the same by SSL? What about write-socket-stream is unique to SSL that it was needed to begin with?

Because a system call sendfile can't send a file over a TLS connection by encrypting it. The latest Linux supports kernel-level TLS, but it's off by default and we need to build it on our own to use it.

[daninus14]

Thanks, I'm just using a fresh ubuntu LTS without nginx or any reverse proxy, just straight woo. I guess if it comes up for anyone else they may help us figure it out