Zig std nonfunctional on Windows after 0.15.1 update

[trypsynth]

Zig Version

0.15.1

Steps to Reproduce and Observed Behavior

1. Install Zig on Windows (I did it via scoop).
2. run zig std from cmd.exe.
3. See this whenever the page tries to load, and an endless loading spinner: error: unable to serve /sources.tar: ConnectionResetByPeer.

Expected Behavior

zig std should work as it always has.

[scottredig]

I don't know if this is related or not, but additionally running zig std puts Command Prompt into a "thinking" animation state, like it's computing something. This is quite distracting for a steady state serving of a webserver, and even after killing the process doesn't go away.

[andrewrk]

missing call to std.Progress.setStatus: https://ziglang.org/download/0.15.1/release-notes.html#stdProgress-supports-progress-bar-escape-codes

[Parzival-3141]

Running with debug enabled nets this panic:

env ZIG_DEBUG_CMD=1 build\stage3\bin\zig.exe std
http://127.0.0.1:25035/
thread 8024 panic: reached unreachable code
C:\zigsrc\zig\lib\std\debug.zig:562:14: 0x7ff74ca592ad in assert (std_zcu.obj)
    if (!ok) unreachable; // assertion failure
             ^
C:\zigsrc\zig\lib\std\http.zig:978:23: 0x7ff74cb0bd0b in chunkedSendFile (std_zcu.obj)
                assert(file_reader.atEnd());
                      ^
C:\zigsrc\zig\lib\std\Io\Writer.zig:896:29: 0x7ff74cb27098 in sendFile (std_zcu.obj)
    return w.vtable.sendFile(w, file_reader, limit);
                            ^
C:\zigsrc\zig\lib\std\Io\Writer.zig:942:27: 0x7ff74cb0d7a6 in sendFileAll (std_zcu.obj)
        const n = sendFile(w, file_reader, .limited(remaining)) catch |err| switch (err) {
                          ^
C:\zigsrc\zig\lib\std\tar\Writer.zig:57:44: 0x7ff74caf2940 in writeFile (std_zcu.obj)
    _ = try w.underlying_writer.sendFileAll(file_reader, .unlimited);
                                           ^
C:\zigsrc\zig\lib\compiler\std-docs.zig:227:31: 0x7ff74caf11e5 in serveSourcesTar (std_zcu.obj)
        try archiver.writeFile(entry.path, &file_reader, stat.mtime);
                              ^
C:\zigsrc\zig\lib\compiler\std-docs.zig:151:28: 0x7ff74caf2e87 in serveRequest (std_zcu.obj)
        try serveSourcesTar(request, context);
                           ^
C:\zigsrc\zig\lib\compiler\std-docs.zig:109:21: 0x7ff74cadee00 in accept (std_zcu.obj)
        serveRequest(&request, context) catch |err| switch (err) {
                    ^
C:\zigsrc\zig\lib\std\Thread.zig:510:13: 0x7ff74cadcab3 in callFn__anon_33976 (std_zcu.obj)
            @call(.auto, f, args);
            ^
C:\zigsrc\zig\lib\std\Thread.zig:623:30: 0x7ff74cad66e2 in entryFn (std_zcu.obj)
                return callFn(f, self.fn_args);
                             ^
???:?:?: 0x7ffc9f487373 in ??? (KERNEL32.DLL)
???:?:?: 0x7ffca149cc90 in ??? (ntdll.dll)
error: the following build command failed with exit code 3:
C:\Users\pocke\AppData\Local\zig\o\5e5f795ae8204308e02272a373d37e3b\std.exe lib C:\zigsrc\zig\build\stage3\bin\zig.exe C:\Users\pocke\AppData\Local\zig

https://github.com/ziglang/zig/blob/bc7955306e3480fc065277d0b6c5abb6797a27ae/lib/std/http.zig#L975-L980 This code doesn't make sense, as the file_reader passed in may not be the original one since this "end of chunk" codepath is only hit by calling the function an additional time, even if you've already logically written the entire file. The chunking logic here needs to be reconsidered.

Note that running the same zig std command as above works fine on linux.

Removing the assert gives the original error, coming from WSASend returning WSAECONNABORTED. Using chrome dev-tools, I'm seeing the browser closing the connection with ERR_INVALID_CHUNKED_ENCODING:

t=117 [st=3]          HTTP_TRANSACTION_READ_RESPONSE_HEADERS
                      --> HTTP/1.1 200 OK
                          transfer-encoding: chunked
                          content-type: application/x-tar
                          cache-control: max-age=0, must-revalidate
t=117 [st=3]       -HTTP_TRANSACTION_READ_HEADERS
t=117 [st=3]        HTTP_CACHE_WRITE_INFO  [dt=0]
t=117 [st=3]        HTTP_CACHE_WRITE_DATA  [dt=0]
t=117 [st=3]        NETWORK_DELEGATE_HEADERS_RECEIVED  [dt=0]
t=117 [st=3]     -URL_REQUEST_START_JOB
t=117 [st=3]      URL_REQUEST_DELEGATE_RESPONSE_STARTED  [dt=1]
t=118 [st=4]      HTTP_TRANSACTION_READ_BODY  [dt=0]
t=118 [st=4]      URL_REQUEST_JOB_FILTERED_BYTES_READ
                  --> byte_count = 3963
t=118 [st=4]      HTTP_TRANSACTION_READ_BODY  [dt=0]
t=118 [st=4]      URL_REQUEST_JOB_FILTERED_BYTES_READ
                  --> byte_count = 65536
t=118 [st=4]      HTTP_TRANSACTION_READ_BODY  [dt=1]
                  --> net_error = -321 (ERR_INVALID_CHUNKED_ENCODING)
t=119 [st=5]      FAILED
                  --> net_error = -321 (ERR_INVALID_CHUNKED_ENCODING)

I haven't been able to narrow the bug down further, but my guess is it's in the chunked encoding implementation resulting in invalid chunks being sent over the wire.

[squeek502]

Looked into this a bit. From what I can tell, there are two issues here:

1. http.chunkedSendFile effectively assumes that sendFile is implemented. If it is not (like on Windows), then the call to sendFileHeader will error with error.Unimplemented, leading to sendFileAll falling back to sendFileReadingAll which does not call into vtable.sendFile and therefore chunkedSendFile is never called again, meaning the terminating \r\n is never written.
2. In scenarios where sendFileHeader is not called (i.e. this out.write branch is executed), file_reader.pos is never updated and therefore the assertion will fail as shown in the previous comment by @Parzival-3141.
   • EDIT: Thinking about it, I think this part is just a symptom of the first. Since the chunk written via sendFileReadingAll is not properly terminated, state.chunk_len is not cleared here and so on the next call of chunkedSendFile it's in an improper state that allows it to write from the buffer and then reach the .chunk_len = 2 state without ever reading from the file.

[mbrock]

I'm not sure it's a problem that sendFileReadingAll without sendfile doesn't go through chunkedSendFile. It has to go through the chunked drain anyway, which is supposed to take care of the chunk framing. But that depends on the correctness of writeSplatHeaderLimit, hence #25360.

[squeek502]

You're right. I do know that the problem still reproduces with the patch from #25360 applied, though, so there's more to figure out.

[squeek502]

I think the problem is that chunkedSendFile isn't returning error.Unimplemented early enough, so it's getting into a weird state where most of a file is being handled by chunkedDrain, but then the leftover bits that remain in the buffer get handled by chunkedSendFile when writing the next file and that makes things go haywire.

Adding

if (builtin.os.tag == .windows) return error.Unimplemented;

to the top of chunkedSendFile does indeed fix it, so I think I'm on the right track. Not sure exactly what the most correct fix should look like yet, though.

[dasimmet]

I looked at this for a while and found that sendFileHeader will write the header downstream even if it returns error.Unimplemented.

not sure if its a good idea to roll back w.end on error but I dont see another option. It seems to fix the encoding issue for me in #25785 : https://github.com/ziglang/zig/pull/25785/commits/6fa655ae4f1f191b464eeaea2c768da3e3003c01

[dasimmet]

The other change i needed was to replace the assertion in case we are at chunk end and continue with a new chunk if the file is not at its end: https://github.com/ziglang/zig/pull/25785/files#diff-0e336c73b568de72cd1961a8642c375af54e42022e6485541245bb4b4e2d4df6R978