zig cc can fail with extremely long argument list

[sin-ack]

Zig Version

0.10.0-dev.699+55fa349ad

Steps to Reproduce

Note that this isn't really a problem for most purposes as nobody should be hitting the ARG_MAX limit in normal use.

1. Generate a very long argument list, check your actual maximum with getconf ARG_MAX (on my system, it's 2097152)
2. Call zig cc with such an argument list, i.e.: zig cc test.c -o test -DAAAAAAAAA...

Expected Behavior

Compile to proceed as usual.

Actual Behavior

info: Usage: zig [command] [options]

Commands:

  build            Build project from build.zig
  init-exe         Initialize a `zig build` application in the cwd
  init-lib         Initialize a `zig build` library in the cwd

  ast-check        Look for simple compile errors in any set of files
  build-exe        Create executable from source or object files
  build-lib        Create library from source or object files
  build-obj        Create object from source or object files
  fmt              Reformat Zig source into canonical form
  run              Create executable and run immediately
  test             Create and run a test build
  translate-c      Convert C code to Zig code

  ar               Use Zig as a drop-in archiver
  cc               Use Zig as a drop-in C compiler
  c++              Use Zig as a drop-in C++ compiler
  dlltool          Use Zig as a drop-in dlltool.exe
  lib              Use Zig as a drop-in lib.exe
  ranlib           Use Zig as a drop-in ranlib

  env              Print lib path, std path, cache directory, and version
  help             Print this help and exit
  libc             Display native libc paths file or validate one
  targets          List available compilation targets
  version          Print version number and exit
  zen              Print Zen of Zig and exit

General Options:

  -h, --help       Print command-specific usage

error: unknown command: @/tmp/response-8e9edb.txt

[sin-ack]

Remarks: This happens because such an argument lists triggers a behavior in Clang to invoke subprograms with clang @/tmp/response-foobar.txt (Clang will actually read the argument list if a file is passed like this). The problem is that argv[0] is zig when the zig binary is used for C compilation, so this fails. I haven't been able to catch the argument list in action yet so I can't say whether it contains the right argument list for zig clang to work or something else.

[schmee]

Duplicate of https://github.com/ziglang/zig/issues/10693?

[sin-ack]

Perhaps, seems like #9654 didn't fix this case.

[emidoots]

This is not a duplicate of #10693.

This may be due to shelling out to lld after the response file is read/expanded by zig cc itself.

It's easier to tell where the cause is on Windows because you get a stacktrace: Create a file foo with omg.c omg.c omg.c omg.c repeated until it's 64 KiB. Then create an empty omg.c file, and run it zig cc "@tmp":

PS C:\Users\slimsag\Desktop\mach2\gpu-dawn> zig cc "@tmp"
error.Unexpected: GetLastError(206): The filename or extension is too long.

C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\debug.zig:531:53: 0x7ff60d2ca8bb in std.debug.writeCurrentStackTraceWindows (zig.obj)
    const n = windows.ntdll.RtlCaptureStackBackTrace(0, addr_buf.len, @ptrCast(**anyopaque, &addr_buf), null);
                                                    ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\debug.zig:511:45: 0x7ff60d008cc6 in std.debug.writeCurrentStackTrace (zig.obj)
        return writeCurrentStackTraceWindows(out_stream, debug_info, tty_config, start_addr);
                                            ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\debug.zig:125:31: 0x7ff60cf265ee in std.debug.dumpCurrentStackTrace (zig.obj)
        writeCurrentStackTrace(stderr, debug_info, detectTTYConfig(), start_addr) catch |err| {
                              ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\os\windows.zig:2026:40: 0x7ff60d23296f in std.os.windows.unexpectedError (zig.obj)
        std.debug.dumpCurrentStackTrace(null);
                                       ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\os\windows.zig:1606:49: 0x7ff60d5b9f4f in std.os.windows.CreateProcessW (zig.obj)
            else => |err| return unexpectedError(err),
                                                ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\child_process.zig:952:34: 0x7ff60d43b509 in std.child_process.windowsCreateProcess (zig.obj)
    return windows.CreateProcessW(
                                 ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\child_process.zig:842:29: 0x7ff60d22fdca in std.child_process.ChildProcess::std.child_process.ChildProcess.spawnWindows (zig.obj)
        windowsCreateProcess(app_path_w.ptr, cmd_line_w.ptr, envp_ptr, cwd_w_ptr, &siStartInfo, &piProcInfo) catch |no_path_err| {
                            ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\child_process.zig:132:37: 0x7ff60d22a55a in std.child_process.ChildProcess::std.child_process.ChildProcess.spawn (zig.obj)
            return self.spawnWindows();
                                    ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\child_process.zig:139:23: 0x7ff60d0289c9 in std.child_process.ChildProcess::std.child_process.ChildProcess.spawnAndWait (zig.obj)
        try self.spawn();
                      ^
C:\Users\slimsag\Desktop\zig\src\link\Coff.zig:1381:48: 0x7ff60d0f97ab in link.Coff.linkWithLLD (zig.obj)
                const term = child.spawnAndWait() catch |err| {
                                               ^
C:\Users\slimsag\Desktop\zig\src\link\Coff.zig:840:32: 0x7ff60d0ef809 in link.Coff.flush (zig.obj)
        return self.linkWithLLD(comp);
                               ^
C:\Users\slimsag\Desktop\zig\src\link.zig:613:70: 0x7ff60d0ec659 in link.File::link.File.flush (zig.obj)
            .coff => return @fieldParentPtr(Coff, "base", base).flush(comp),
                                                                     ^
C:\Users\slimsag\Desktop\zig\src\Compilation.zig:2187:28: 0x7ff60d0b1ef2 in Compilation.flush (zig.obj)
    try comp.bin_file.flush(comp); // This is needed before reading the error flags.
                           ^
C:\Users\slimsag\Desktop\zig\src\Compilation.zig:2158:27: 0x7ff60d0a86b4 in Compilation.update (zig.obj)
            try comp.flush();
                          ^
C:\Users\slimsag\Desktop\zig\src\main.zig:3048:20: 0x7ff60d02609c in main.updateModule (zig.obj)

                   ^
C:\Users\slimsag\Desktop\zig\src\main.zig:2735:17: 0x7ff60cf3d684 in main.buildOutputType (zig.obj)
    };
                ^
C:\Users\slimsag\Desktop\zig\src\main.zig:226:31: 0x7ff60cf20116 in main.mainArgs (zig.obj)
        return buildOutputType(gpa, arena, args, .cc);
                              ^
C:\Users\slimsag\Desktop\zig\src\stage1.zig:48:24: 0x7ff60cf1f88d in main (zig.obj)
        stage2.mainArgs(gpa, arena, args) catch unreachable;
                       ^
Unable to dump stack trace: InvalidDebugInfo
error(link): unable to spawn C:\Users\slimsag\Desktop\zig\zig-out\bin\zig.exe: Unexpected
thread 14620 panic: attempt to unwrap error: UnableToSpawnSelf
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\child_process.zig:139:9: 0x7ff60d0289ec in std.child_process.ChildProcess::std.child_process.ChildProcess.spawnAndWait (zig.obj)
        try self.spawn();
        ^
C:\Users\slimsag\Desktop\zig\src\link\Coff.zig:1383:21: 0x7ff60d0f981a in link.Coff.linkWithLLD (zig.obj)
                    return error.UnableToSpawnSelf;
                    ^
C:\Users\slimsag\Desktop\zig\src\link\Coff.zig:840:9: 0x7ff60d0ef81c in link.Coff.flush (zig.obj)
        return self.linkWithLLD(comp);
        ^
C:\Users\slimsag\Desktop\zig\src\link.zig:613:22: 0x7ff60d0ec66c in link.File::link.File.flush (zig.obj)
            .coff => return @fieldParentPtr(Coff, "base", base).flush(comp),
                     ^
C:\Users\slimsag\Desktop\zig\src\Compilation.zig:2187:5: 0x7ff60d0b1f11 in Compilation.flush (zig.obj)
    try comp.bin_file.flush(comp); // This is needed before reading the error flags.
    ^
C:\Users\slimsag\Desktop\zig\src\Compilation.zig:2158:13: 0x7ff60d0a876f in Compilation.update (zig.obj)
            try comp.flush();
            ^
C:\Users\slimsag\Desktop\zig\src\main.zig:3048:5: 0x7ff60d0260c4 in main.updateModule (zig.obj)

    ^
C:\Users\slimsag\Desktop\zig\src\main.zig:2737:21: 0x7ff60cf3d8e9 in main.buildOutputType (zig.obj)
    updateModule(gpa, comp, hook) catch |err| switch (err) {
                    ^
C:\Users\slimsag\Desktop\zig\src\main.zig:226:9: 0x7ff60cf20139 in main.mainArgs (zig.obj)
        return buildOutputType(gpa, arena, args, .cc);
        ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\os.zig:2381:9: 0x7ff60cfb7998 in std.os.mkdirat (zig.obj)
        return mkdiratW(dir_fd, sub_dir_path_w.span(), mode);
        ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\fs.zig:1270:9: 0x7ff60cfb788a in std.fs.Dir::std.fs.Dir.makeDir (zig.obj)
        try os.mkdirat(self.fd, sub_path, default_new_dir_mode);
        ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\os\windows.zig:136:35: 0x7ff60d29040b in std.os.windows.OpenFile (zig.obj)
        .OBJECT_NAME_COLLISION => return error.PathAlreadyExists,
                                  ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\os.zig:2450:21: 0x7ff60cfb7aa0 in std.os.mkdiratW (zig.obj)
        else => |e| return e,
                    ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\os.zig:2381:9: 0x7ff60cfb7998 in std.os.mkdirat (zig.obj)
        return mkdiratW(dir_fd, sub_dir_path_w.span(), mode);
        ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\fs.zig:1270:9: 0x7ff60cfb788a in std.fs.Dir::std.fs.Dir.makeDir (zig.obj)
        try os.mkdirat(self.fd, sub_path, default_new_dir_mode);
        ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\os\windows.zig:136:35: 0x7ff60d29040b in std.os.windows.OpenFile (zig.obj)
        .OBJECT_NAME_COLLISION => return error.PathAlreadyExists,
                                  ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\os.zig:2450:21: 0x7ff60cfb7aa0 in std.os.mkdiratW (zig.obj)
        else => |e| return e,
                    ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\os.zig:2381:9: 0x7ff60cfb7998 in std.os.mkdirat (zig.obj)
        return mkdiratW(dir_fd, sub_dir_path_w.span(), mode);
        ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\fs.zig:1270:9: 0x7ff60cfb788a in std.fs.Dir::std.fs.Dir.makeDir (zig.obj)
        try os.mkdirat(self.fd, sub_path, default_new_dir_mode);
        ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\os\windows.zig:136:35: 0x7ff60d29040b in std.os.windows.OpenFile (zig.obj)
        .OBJECT_NAME_COLLISION => return error.PathAlreadyExists,
                                  ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\os.zig:2450:21: 0x7ff60cfb7aa0 in std.os.mkdiratW (zig.obj)
        else => |e| return e,
                    ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\os.zig:2381:9: 0x7ff60cfb7998 in std.os.mkdirat (zig.obj)
        return mkdiratW(dir_fd, sub_dir_path_w.span(), mode);
        ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\fs.zig:1270:9: 0x7ff60cfb788a in std.fs.Dir::std.fs.Dir.makeDir (zig.obj)
        try os.mkdirat(self.fd, sub_path, default_new_dir_mode);
        ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\os\windows.zig:136:35: 0x7ff60d29040b in std.os.windows.OpenFile (zig.obj)
        .OBJECT_NAME_COLLISION => return error.PathAlreadyExists,
                                  ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\os.zig:2450:21: 0x7ff60cfb7aa0 in std.os.mkdiratW (zig.obj)
        else => |e| return e,
                    ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\os.zig:2381:9: 0x7ff60cfb7998 in std.os.mkdirat (zig.obj)
        return mkdiratW(dir_fd, sub_dir_path_w.span(), mode);
        ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\fs.zig:1270:9: 0x7ff60cfb788a in std.fs.Dir::std.fs.Dir.makeDir (zig.obj)
        try os.mkdirat(self.fd, sub_path, default_new_dir_mode);
        ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\os\windows.zig:2028:5: 0x7ff60d23297e in std.os.windows.unexpectedError (zig.obj)
    return error.Unexpected;
    ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\os\windows.zig:1606:27: 0x7ff60d5b9f5c in std.os.windows.CreateProcessW (zig.obj)
            else => |err| return unexpectedError(err),
                          ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\child_process.zig:952:5: 0x7ff60d43b51c in std.child_process.windowsCreateProcess (zig.obj)
    return windows.CreateProcessW(
    ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\child_process.zig:843:52: 0x7ff60d23001d in std.child_process.ChildProcess::std.child_process.ChildProcess.spawnWindows (zig.obj)
            if (no_path_err != error.FileNotFound) return no_path_err;
                                                   ^
C:\zig-windows-x86_64-0.10.0-dev.862+539bb8a2d\lib\std\child_process.zig:132:13: 0x7ff60d22a56d in std.child_process.ChildProcess::std.child_process.ChildProcess.spawn (zig.obj)
            return self.spawnWindows();
            ^
Unable to dump stack trace: MissingDebugInfo

[jcmoyer]

I believe I ran into this on windows while trying to compile fftw with cmake using CMAKE_C_COMPILER=zig cc. It fails on the final link:

D:\Scratch\fftw-3.3.10\build>zig cc -DWITH_OUR_MALLOC -O3 -DNDEBUG   -shared -o Release\libfftw3f.dll -Wl,--out-implib,Release\libfftw3f.dll.a -Wl,--major-image-version,3,--minor-image-version,0 @CMakeFiles\fftw3f.Release.rsp
error(link): unable to spawn D:\Scratch\zig-install\bin\zig.exe: Unexpected
error: UnableToSpawnSelf
ninja: build stopped: subcommand failed.

fftw3f.Release.rsp is 46K. zig 0.10.0-dev.1516+b74f29241

I was able to work around the problem by disabling library features until cmake stopped using a response file.

[calavera]

I bumped into this problem too on a Windows build, and I really have no idea how to reproduce it, because it only happens in CI. If anyone is interested in taking a look to see if there is a clear fix, this is the full log:

https://ci.appveyor.com/project/AWSSAMCLI/aws-lambda-builders/builds/43252407/job/9n3ebh8fb8khkjw8

[lost22git]

Same problem

[pkova]

Fixed by #19850

[calavera]

Is the fix going to be in 0.13.0, or do you have plans to include it in a 0.12.1 version at some point?

[calavera]

I've confirmed that this issue doesn't happen on 0.13.0 with the release of https://github.com/ziglang/zig/pull/19850 🎊

@andrewrk I think this issue can be closed as resolved.