zig build: panic while downloading package

[nektro]

Zig Version

0.11.0-dev.2986+012f9a97e

Steps to Reproduce and Observed Behavior

$ git clone https://codeberg.org/hanna/mocha
$ cd mocha
$ git checkout 284e6dc9af3fdae262d66d8390d81c8a11021a1e
$ zig build test -fsummary

thread 837374 panic: @memcpy arguments have non-equal lengths
/home/meghan/src/zig/lib/std/crypto/tls/Client.zig:1062:25: 0x15ea9c5 in readvAdvanced__anon_149203 (zig)
            @memcpy(frag[0..in], first);
                        ^
/home/meghan/src/zig/lib/std/crypto/tls/Client.zig:898:53: 0x15e81bb in readvAtLeast__anon_149202 (zig)
        var amt = try c.readvAdvanced(stream, iovecs[vec_i..]);
                                                    ^
/home/meghan/src/zig/lib/std/crypto/tls/Client.zig:859:24: 0x15e7f5f in readAtLeast__anon_149201 (zig)
    return readvAtLeast(c, stream, &iovecs, len);
                       ^
/home/meghan/src/zig/lib/std/crypto/tls/Client.zig:864:23: 0x15e7ea1 in read__anon_149200 (zig)
    return readAtLeast(c, stream, buffer, 1);
                      ^
/home/meghan/src/zig/lib/std/http/Client.zig:173:46: 0x10d5501 in read (zig)
            .tls => conn.tls_client.read(conn.stream, buffer),
                                             ^
/home/meghan/src/zig/lib/std/http/Client.zig:296:46: 0x10d5e36 in readAtLeast (zig)
                return bconn.conn.read(buffer[out_index..]);
                                             ^
/home/meghan/src/zig/lib/std/http/Client.zig:306:33: 0xe3cee7 in read (zig)
        return bconn.readAtLeast(buffer, 1);
                                ^
/home/meghan/src/zig/lib/std/http/protocol.zig:586:73: 0xe3c6d2 in read__anon_103355 (zig)
                        const nread = try bconn.read(buffer[out_index..][0..can_read]);
                                                                        ^
/home/meghan/src/zig/lib/std/http/Client.zig:663:111: 0xe3d976 in transferRead (zig)
            const amt = try req.response.parser.read(&req.connection.data.buffered, buf[index..], req.response.skip);
                                                                                                              ^
/home/meghan/src/zig/lib/std/http/Client.zig:788:41: 0xe44b33 in read (zig)
            else => try req.transferRead(buffer),
                                        ^
/home/meghan/src/zig/lib/std/io/reader.zig:27:31: 0xe46308 in read (zig)
            return readFn(self.context, buffer);
                              ^
/home/meghan/src/zig/lib/std/io/buffered_reader.zig:27:71: 0xe460c6 in read (zig)
                    const n = try self.unbuffered_reader.read(self.buf[0..]);
                                                                      ^
/home/meghan/src/zig/lib/std/io/reader.zig:27:31: 0xe46518 in read (zig)
            return readFn(self.context, buffer);
                              ^
/home/meghan/src/zig/lib/std/io/reader.zig:243:50: 0x10f0fe7 in readByte (zig)
            const amt_read = try self.read(result[0..]);
                                                 ^
/home/meghan/src/zig/lib/std/compress/deflate/decompressor.zig:848:55: 0x10f17f2 in huffSym (zig)
                    var c = self.inner_reader.readByte() catch |e| {
                                                      ^
/home/meghan/src/zig/lib/std/compress/deflate/decompressor.zig:621:57: 0xe48a20 in huffmanBlock (zig)
                        var v = try self.huffSym(self.hl.?);
                                                        ^
/home/meghan/src/zig/lib/std/compress/deflate/decompressor.zig:437:42: 0xc10be0 in nextBlock (zig)
                    try self.huffmanBlock();
                                         ^
/home/meghan/src/zig/lib/std/compress/deflate/decompressor.zig:471:26: 0xe4bb29 in read (zig)
                self.step(self) catch |e| {
                         ^
/home/meghan/src/zig/lib/std/compress/gzip.zig:128:45: 0xe4b243 in read (zig)
            const r = try self.inflater.read(buffer);
                                            ^
/home/meghan/src/zig/lib/std/io/reader.zig:27:31: 0xe4bc18 in read (zig)
            return readFn(self.context, buffer);
                              ^
/home/meghan/src/zig/lib/std/io/reader.zig:46:49: 0xc13662 in readAtLeast (zig)
                const amt = try self.read(buffer[index..]);
                                                ^
/home/meghan/src/zig/lib/std/tar.zig:110:45: 0xc12285 in pipeToFileSystem__anon_49850 (zig)
        end += try reader.readAtLeast(buffer[end..], ask_header);
                                            ^
/home/meghan/src/zig/src/Package.zig:571:60: 0xc138d2 in unpackTarball__anon_49616 (zig)
    try std.tar.pipeToFileSystem(out_dir, decompress.reader(), .{
                                                           ^
/home/meghan/src/zig/src/Package.zig:500:55: 0xc26187 in fetchAndUnpack (zig)
            try unpackTarball(gpa, &req, tmp_directory.handle, std.compress.gzip);
                                                      ^
/home/meghan/src/zig/src/Package.zig:283:43: 0xc2cd50 in fetchAndAddDependencies (zig)
        const sub_pkg = try fetchAndUnpack(
                                          ^
/home/meghan/src/zig/src/main.zig:4399:67: 0xa86398 in cmdBuild (zig)
            const fetch_result = build_pkg.fetchAndAddDependencies(
                                                                  ^
/home/meghan/src/zig/src/main.zig:298:24: 0xa52bfd in mainArgs (zig)
        return cmdBuild(gpa, arena, cmd_args);
                       ^
/home/meghan/src/zig/src/main.zig:211:20: 0xa51725 in main (zig)
    return mainArgs(gpa, arena, args);
                   ^
/home/meghan/src/zig/lib/std/start.zig:609:37: 0xa53da8 in main (zig)
            const result = root.main() catch |err| {
                                    ^
???:?:?: 0x7fb7eb06e24d in ??? (???)
???:?:?: 0x7ffd0dc665db in ??? (???)
Aborted (core dumped)

Expected Behavior

hanna/mocha tests to be run

[hqnna]

Can you try it on https://github.com/hqnna/mocha? the codebase is more updated, though I had the same error.

[nektro]

same error as of https://github.com/hqnna/mocha/commit/d31a29a7dbda5c01e34799b1e63b4c17048bfec3

[andrewrk]

Thank you for the nice stack trace, fix coming along shortly.

[andrewrk]

with

--- a/lib/std/crypto/tls/Client.zig
+++ b/lib/std/crypto/tls/Client.zig
@@ -1059,7 +1059,7 @@ pub fn readvAdvanced(c: *Client, stream: anytype, iovecs: []const std.os.iovec)
             if (frag1.len < second_len)
                 return finishRead2(c, first, frag1, vp.total);
 
-            @memcpy(frag[0..in], first);
+            @memcpy(frag[0..first.len], first);
             @memcpy(frag[first.len..][0..second_len], frag1[0..second_len]);
             frag = frag[0..full_record_len];
             frag1 = frag1[second_len..];

now I'm getting

thread 364623 panic: index out of bounds: index 1370, len 1233
/home/andy/dev/zig/lib/std/crypto/tls/Client.zig:1180:45: 0x6669c5a in readvAdvanced__anon_149320 (zig)
                                @memcpy(dest[0..msg.len], msg);
                                            ^
/home/andy/dev/zig/lib/std/crypto/tls/Client.zig:898:53: 0x66624ab in readvAtLeast__anon_149319 (zig)
        var amt = try c.readvAdvanced(stream, iovecs[vec_i..]);
                                                    ^
/home/andy/dev/zig/lib/std/crypto/tls/Client.zig:859:24: 0x666224f in readAtLeast__anon_149318 (zig)
    return readvAtLeast(c, stream, &iovecs, len);
                       ^
/home/andy/dev/zig/lib/std/crypto/tls/Client.zig:864:23: 0x6662191 in read__anon_149317 (zig)
    return readAtLeast(c, stream, buffer, 1);
                      ^
/home/andy/dev/zig/lib/std/http/Client.zig:173:46: 0x614fc21 in read (zig)
            .tls => conn.tls_client.read(conn.stream, buffer),
                                             ^

Thanks for the report, I'll continue to work on this.

[andrewrk]

Btw @truemedian, the TLS code can be more efficient if a larger output buffer is given to the read function:

https://github.com/ziglang/zig/blob/d70853ba39cb1694d91a763ea8492f3aed1494a7/lib/std/crypto/tls/Client.zig#L1171-L1196

Currently the top branch is executing, but the happy path is the bottom one.

[weejulius]

Is there any wordaround for this issue?

[renerocksai]

It's back. My CI builds have been failing, and I'm experiencing it on my office machine, too which doesn't have the dependency cached:

[kamidev]

I get exactly the same problem building a local sample app for the Zap library. The same app downloaded dependent packages and built correctly a few days ago, using an earlier zig version.

[leecannon]

ZLS is also affected by this again.

[DraagrenKirneh]

Maybe it is worth to change the ReadError in Client.zig to also include the more fine grained TLS errors instead of generalizing them to TlsFailure? It would help to pinpoint a bit better what the cause is, and could be changed back again once we see that errors have been fixed.

[renerocksai]

i'll de-dupe again :-) by closing the newly opened issues

[truemedian]

What would be more helpful here is the error return trace, which should include where the original error originated.

[DraagrenKirneh]

Here is a trace from a debug build for BadReaderState.

error: BadReaderState
C:\Git\zigcode\zig\lib\std\compress\deflate\decompressor.zig:461:25: 0x7ff6b9903935 in read (zig.exe.obj)
                        return self.err.?;
                        ^
C:\Git\zigcode\zig\lib\std\compress\gzip.zig:128:23: 0x7ff6b990329f in read (zig.exe.obj)
            const r = try self.inflater.read(buffer);
                      ^
C:\Git\zigcode\zig\lib\std\io\reader.zig:27:13: 0x7ff6b9903c39 in read (zig.exe.obj)
            return readFn(self.context, buffer);
            ^
C:\Git\zigcode\zig\lib\std\io\reader.zig:46:29: 0x7ff6b96de8b8 in readAtLeast (zig.exe.obj)
                const amt = try self.read(buffer[index..]);
                            ^
C:\Git\zigcode\zig\lib\std\tar.zig:110:16: 0x7ff6b96dd4ab in pipeToFileSystem__anon_50286 (zig.exe.obj)
        end += try reader.readAtLeast(buffer[end..], ask_header);
               ^
C:\Git\zigcode\zig\src\Package.zig:602:5: 0x7ff6b96deb1f in unpackTarball__anon_50054 (zig.exe.obj)
    try std.tar.pipeToFileSystem(out_dir, decompress.reader(), .{
    ^
C:\Git\zigcode\zig\src\Package.zig:519:13: 0x7ff6b96e83f2 in fetchAndUnpack (zig.exe.obj)
            try unpackTarball(gpa, &req, tmp_directory.handle, std.compress.gzip);
            ^
C:\Git\zigcode\zig\src\Package.zig:282:21: 0x7ff6b96f0d37 in fetchAndAddDependencies (zig.exe.obj)
        const sub = try fetchAndUnpack(
                    ^
C:\Git\zigcode\zig\src\main.zig:4463:13: 0x7ff6b954122d in cmdBuild (zig.exe.obj)
            try fetch_result;
            ^
C:\Git\zigcode\zig\src\main.zig:285:9: 0x7ff6b950c879 in mainArgs (zig.exe.obj)
        return cmdBuild(gpa, arena, cmd_args);
        ^
C:\Git\zigcode\zig\src\main.zig:213:5: 0x7ff6b950be28 in main (zig.exe.obj)
    return mainArgs(gpa, arena, args);

And for TlsInitializationFailed

PS C:\Git\zigcode\tmp\xyz> C:\Git\zigcode\zig\build\stage3\bin\zig.exe build
error: TlsInitializationFailed
C:\Git\zigcode\zig\lib\std\crypto\chacha20.zig:717:17: 0x7ff6ba049de8 in decrypt (zig.exe.obj)
                return error.AuthenticationFailed;
                ^
C:\Git\zigcode\zig\lib\std\crypto\tls\Client.zig:462:29: 0x7ff6ba0f1e28 in init__anon_150939 (zig.exe.obj)
                            return error.TlsBadRecordMac;
                            ^
C:\Git\zigcode\zig\lib\std\http\Client.zig:986:117: 0x7ff6b9bbf663 in connectUnproxied (zig.exe.obj)
            conn.data.buffered.conn.tls_client.* = std.crypto.tls.Client.init(stream, client.ca_bundle, host) catch return error.TlsInitializationFailed;
                                                                                                                    ^
C:\Git\zigcode\zig\lib\std\http\Client.zig:1021:9: 0x7ff6b98f4f08 in connect (zig.exe.obj)
        return client.connectUnproxied(host, port, protocol);
        ^
C:\Git\zigcode\zig\lib\std\http\Client.zig:1086:44: 0x7ff6b96ea5e4 in request (zig.exe.obj)
    const conn = options.connection orelse try client.connect(host, port, protocol);
                                           ^
C:\Git\zigcode\zig\src\Package.zig:497:19: 0x7ff6b96e7d50 in fetchAndUnpack (zig.exe.obj)
        var req = try http_client.request(.GET, uri, h, .{});
                  ^
C:\Git\zigcode\zig\src\Package.zig:282:21: 0x7ff6b96f0d37 in fetchAndAddDependencies (zig.exe.obj)
        const sub = try fetchAndUnpack(
                    ^
C:\Git\zigcode\zig\src\main.zig:4463:13: 0x7ff6b954122d in cmdBuild (zig.exe.obj)
            try fetch_result;
            ^
C:\Git\zigcode\zig\src\main.zig:285:9: 0x7ff6b950c879 in mainArgs (zig.exe.obj)
        return cmdBuild(gpa, arena, cmd_args);
        ^
C:\Git\zigcode\zig\src\main.zig:213:5: 0x7ff6b950be28 in main (zig.exe.obj)
    return mainArgs(gpa, arena, args);

Also got this one:

C:\Git\zigcode\zig\build\stage3\bin\zig.exe build
thread 6324 panic: integer overflow
C:\Git\zigcode\zig\lib\std\linked_list.zig:293:22: 0x7ff6b9bc82a6 in remove (zig.exe.obj)
            list.len -= 1;
                     ^
C:\Git\zigcode\zig\lib\std\http\Client.zig:105:25: 0x7ff6b98f5d11 in release (zig.exe.obj)
        pool.used.remove(node);
                        ^
C:\Git\zigcode\zig\lib\std\http\Client.zig:563:59: 0x7ff6b96ebdbf in deinit (zig.exe.obj)
        req.client.connection_pool.release(req.client, req.connection);
                                                          ^
C:\Git\zigcode\zig\src\Package.zig:498:25: 0x7ff6b96e7eae in fetchAndUnpack (zig.exe.obj)
        defer req.deinit();
                        ^
C:\Git\zigcode\zig\src\Package.zig:282:39: 0x7ff6b96f0cf0 in fetchAndAddDependencies (zig.exe.obj)
        const sub = try fetchAndUnpack(
                                      ^
C:\Git\zigcode\zig\src\main.zig:4443:67: 0x7ff6b9540e2e in cmdBuild (zig.exe.obj)
            const fetch_result = build_pkg.fetchAndAddDependencies(
                                                                  ^
C:\Git\zigcode\zig\src\main.zig:285:24: 0x7ff6b950c7f6 in mainArgs (zig.exe.obj)
        return cmdBuild(gpa, arena, cmd_args);
                       ^
C:\Git\zigcode\zig\src\main.zig:213:20: 0x7ff6b950bdde in main (zig.exe.obj)
    return mainArgs(gpa, arena, args);
                   ^
C:\Git\zigcode\zig\lib\std\start.zig:508:80: 0x7ff6b950dd21 in main (zig.exe.obj)
    return @call(.always_inline, callMainWithArgs, .{ @intCast(usize, c_argc), @ptrCast([*][*:0]u8, c_argv), envp });
                                                                               ^
???:?:?: 0x7ff6bb46f415 in ??? (zig.exe)
???:?:?: 0x7ff6bb46f47b in ??? (zig.exe)
???:?:?: 0x7ffd7e467613 in ??? (KERNEL32.DLL)
???:?:?: 0x7ffd801826a0 in ??? (ntdll.dll)

TlsAlert

error: TlsAlert
C:\Git\zigcode\zig\lib\std\crypto\tls\Client.zig:1118:25: 0x7ff6ba044144 in readvAdvanced__anon_150827 (zig.exe.obj)
                        return error.TlsAlert;
                        ^
C:\Git\zigcode\zig\lib\std\crypto\tls\Client.zig:898:19: 0x7ff6ba03e12c in readvAtLeast__anon_150826 (zig.exe.obj)
        var amt = try c.readvAdvanced(stream, iovecs[vec_i..]);
                  ^
C:\Git\zigcode\zig\lib\std\crypto\tls\Client.zig:859:5: 0x7ff6ba03def6 in readAtLeast__anon_150825 (zig.exe.obj)
    return readvAtLeast(c, stream, &iovecs, len);
    ^
C:\Git\zigcode\zig\lib\std\crypto\tls\Client.zig:864:5: 0x7ff6ba03de0a in read__anon_150824 (zig.exe.obj)
    return readAtLeast(c, stream, buffer, 1);
    ^
C:\Git\zigcode\zig\lib\std\http\Client.zig:182:31: 0x7ff6b9bbde8c in read (zig.exe.obj)
            error.TlsAlert => return error.TlsAlert,
                              ^
C:\Git\zigcode\zig\lib\std\http\Client.zig:286:23: 0x7ff6b98f3a84 in fill (zig.exe.obj)
        const nread = try bconn.conn.read(bconn.read_buf[0..]);
                      ^
C:\Git\zigcode\zig\lib\std\http\Client.zig:719:17: 0x7ff6b96ebeff in wait (zig.exe.obj)
                try req.connection.data.buffered.fill();
                ^
C:\Git\zigcode\zig\src\Package.zig:501:9: 0x7ff6b96e7ee7 in fetchAndUnpack (zig.exe.obj)
        try req.wait();
        ^
C:\Git\zigcode\zig\src\Package.zig:282:21: 0x7ff6b96f0d37 in fetchAndAddDependencies (zig.exe.obj)
        const sub = try fetchAndUnpack(
                    ^
C:\Git\zigcode\zig\src\main.zig:4463:13: 0x7ff6b954122d in cmdBuild (zig.exe.obj)
            try fetch_result;
            ^
C:\Git\zigcode\zig\src\main.zig:285:9: 0x7ff6b950c879 in mainArgs (zig.exe.obj)
        return cmdBuild(gpa, arena, cmd_args);
        ^
C:\Git\zigcode\zig\src\main.zig:213:5: 0x7ff6b950be28 in main (zig.exe.obj)
    return mainArgs(gpa, arena, args);

[DraagrenKirneh]

Looks like this is now fixed in latest master by: https://github.com/ziglang/zig/commit/ab37ab33ce94b4fb6536bcc2f3981c0cc257c9f0

With it package download works, If i revert it I get BadReaderState etc.

[renerocksai]

I cannot confirm that. The current downloadable zig master is 0.11.0-dev.3309+8f5f8090c. The commit you're referencing is four days old, so it should safely have been part of the nightly build for a few days. As of now, the errors are still there. You might just have been lucky.

My current output shows them all:

zig version
  shell: /usr/bin/bash -e {0}
0.11.0-dev.3309+8f5f8090c

-------------------------------------------
hello
error: BadReaderState
-------------------------------------------
hello2
error: TlsFailure
-------------------------------------------
routes
error: TlsFailure
-------------------------------------------
serve
error: BadReaderState
-------------------------------------------
hello_json
error: TlsInitializationFailed
-------------------------------------------
endpoint
error: TlsAlert
-------------------------------------------
wrk
error: BadReaderState
-------------------------------------------
wrk_zigstd
error: BadReaderState
-------------------------------------------
mustache
./build_all.sh: line 2:  173[5](https://github.com/zigzap/zap/actions/runs/5100665554/jobs/9169079856#step:5:6) Segmentation fault      (core dumped) zig build $i $*
-------------------------------------------
endpoint_auth
error: BadReaderState
-------------------------------------------
http_params
error: BadReaderState
-------------------------------------------
pkghash
error: TlsFailure
-------------------------------------------
cookies
error: BadReaderState
-------------------------------------------
websockets
./build_all.sh: line 2:  1[7](https://github.com/zigzap/zap/actions/runs/5100665554/jobs/9169079856#step:5:8)53 Segmentation fault      (core dumped) zig build $i $*
-------------------------------------------
userpass_session
./build_all.sh: line 2:  1757 Segmentation fault      (core dumped) zig build $i $*
-------------------------------------------
sendfile
error: BadReaderState
-------------------------------------------
middleware
error: TlsAlert
-------------------------------------------
middleware_with_endpoint
error: TlsAlert
-------------------------------------------
senderror
error: TlsFailure
-------------------------------------------
announceybot
error: BadReaderState

[DraagrenKirneh]

I managed to link the wrong commit... The correct one that fixes it is https://github.com/ziglang/zig/commit/ab37ab33ce94b4fb6536bcc2f3981c0cc257c9f0

cc: @renerocksai

[renerocksai]

Oh cool, thx for pointing that out. The currently downloadable version is 3 commits behind that one. So hopes are up again :smile:!

[kamidev]

I built zig after this commit and cannot reproduce the problem anymore. @renerocksai: zap works as expected again, too.

[star-tek-mb]

Errors still exist for me, unfortunately. 0.11.0-dev.3312+ab37ab33c

[hqnna]

unable to reproduce on my machine with commit ab37ab33c

[edyu]

I still have sigsegv using 0.11.0-dev.3312+ab37ab33c

[ikskuh]

Having a TlsInitializationFailed with https://git.random-projects.net/DerTeufel/fraxinus/archive/88fac1ffc627ce5b4e69b0897df93cbb4be5de09.tar.gz (or any other url on my domains)

[truemedian]

git.random-projects.net doesn't support TLS 1.3, which is the only version zig has support for at the moment.

[ikskuh]

git.random-projects.net doesn't support TLS 1.3, which is the only version zig has support for at the moment.

Thanks for clarification. I have to figure out how to enable TLS 1.3, in theory it's active but it seems like nginx doesn't like it right now

[andrewrk]

Original bug was resolved. What remains is a duplicate of #14174.