msys2-autobuild

msys2-autobuild copied to clipboard

Just so we have an issue to link to and discuss maybe

I believe this is a bug in cygwin/msys2-runtime, because it also happened with pacman frequently when it would verify sync db signatures. The workaround I had for that was to append DatabaseNever to SigLevel.

I wonder whether this is still the case for v3.4.*...

This hang seems to be happening much more often with the new 2023 dev kit machine compared to the qc710. It is now even happening when validating signatures on packages, where it would usually only happen validating database signatures before.

Me and the GIMP project have also seen this. There's a comment lost in a Merge Request (Gitlab) somewhere about it. They decided to stop pacman update as part of the builds and the runners are now doing this on a daily scheduled task with timeout/retry overnight :(

I gathered some information that may be helpful for analyzing this issue, and wrote it down here.

I wonder if it might possibly be https://cygwin.com/pipermail/cygwin/2024-February/255431.html. Maybe we can try backporting that patch (msys2/msys2-runtime@4e77fa9b8bf4) and see if the issues go away?

If anyone else wants to try, I built msys2-runtime and msys2-runtime-3.3 with that patch applied in https://github.com/jeremyd2019/MSYS2-packages/actions/runs/7921543265. I am planning to try some things with it and see what happens.

UPDATE: that seems pretty broken. I'm guessing I didn't backport the fix correctly.

https://github.com/jeremyd2019/MSYS2-packages/actions/runs/7924206550 is at least not as immediately broken :wink:. Will test that

I built both 3.4 and 3.3, and 3.3 for 32-bit (which took some doing because any binutils later than 2.40 resulted in a broken msys-2.0.dll). I then set both a Windows 11 VM on the Dev Kit and a Windows 10 install on a Raspberry Pi 4 in a loop running pacman (without disabling db signature checking). The raspberry pi did hang up, but the debugger looks different than I remember. The dev kit vm is still going at last check.

I think the 32-bit on the raspberry pi hung up in pinfo::release calling CloseHandle. Not entirely sure which handle. I'd probably try getting a debug build next, but not sure I care that much about 32-bit (just that I remembered the raspberry pi having hang issues more frequently so wanted to test it too)

Looking back at the cygwin thread, it seems that patch was introduced after a report of a hang with 100% CPU usage, rather than the hang with 0 CPU usage that we see, so I'm not sure it's the same issue. I guess I'll keep looking into the pinfo::release hang I see on the raspberry pi as I have time.

With debug build it hung up somewhere different, but doesn't make any more sense. This time it hung up apparently during process teardown, having called _exit(0), eventually getting to proc_terminate, and is hanging calling TerminateThread on what seems to be a valid handle to a thread. I don't see that thread currently running in the debugger though. I don't see any reason why TerminateThread should hang.

The 64-bit msys2 on windows 11 did eventually hang too ☹️ looking at some of the other messages in the thread I wasn't too positive about that being the issue we were seeing.

I finally have some good news. While I am not even close to a fix, I have a work-around: https://github.com/msys2/MSYS2-packages/pull/4583

Here is a run of Git for Windows' sync job that tries to update, commit & push the git-sdk-arm64 repository. Previously it consistently ran into those hangs, and replacing the pacman.exe built in that PR works around those hangs.

By manually observing the hangs (RDPing into those self-hosted runners) I figured out that there were typically around half a dozen hanging processes whose command-lines were identical to their respective parent processes' command-lines. I've tracked that down to libgpgme's _gpgme_io_spawn() function, which calls a fork() that is immediately followed by another fork() in its child process. GPGME's source code history calls this a "double-fork", and it is not really clear to me why this would be needed.

One thing that helped me tremendously while debugging this was the insight that calling that PowerShell script that runs pacman several times (to do both core and system update, followed by another pacman invocation to allow for Git for Windows' hack that is the post-install script of the git-extra package) works when run in a PowerShell window, but consistently hangs when being run in an SSH session. Or, for that matter, a VS Code terminal (which I ran, for convenience, via a VS Code tunnel).

So these are my thoughts how to proceed from here:

1. Deploying https://github.com/msys2/MSYS2-packages/pull/4583 should work around those Windows/ARM64 hangs.
2. We still need to dig into this a bit further, to actually understand what is going wrong.
3. It might be a Windows bug (or the promised Windows update might be just another work-around), we will see...
4. The work-around I propose points to the double-fork in GPGME as triggering the issue. It won't be trivial, and it will take time, but I am confident, at least, that this can be turned into a small and simple reproducer.
5. My hunch is that it is not a Windows bug, but that the pseudo terminal/pseudo console code in the MSYS2 runtime that runs every time a new process is forked is still not race-free.
6. There are other issues in this area of the MSYS2 runtime that might be related, and need to be addressed even if they are not related. For example, @tyan0 said that crashing sub-processes would likely lead to hangs, too.

In that case, I wonder if there's a race between starting up the wait thread and shutting it down during process exit. Assuming the second fork is followed by an exec in the (grand)child, that could further complicate things because I think that there is some magic that shuffles around the process tree to try to make it look as though exec actually replaced the process instead of starting a new one. (I think that may even be involved in the wait thread).

I never did get a good understanding of locking around this code, either. This is why I was trying Interlocked operations, to see if maybe there was a race going on, because I was seeing things in the debugger like handles that were NULL in the struct, but the stack showed a non-NULL handle passed to functions like CloseHandle or TerminateThread. I think I was satisfied that they were moving the handle into a temp variable and nulling it in the struct before closing it, but it felt like it was trying to avoid a race in a not-horribly-effective manner.

As for a Windows bug, I couldn't see any good reason for TerminateThread to block. I was a little concerned that maybe terminating a thread could leave the emulation in a bad state.

I've tracked that down to libgpgme's _gpgme_io_spawn() function, which calls a fork() that is immediately followed by another fork() in its child process. GPGME's source code history calls this a "double-fork", and it is not really clear to me why this would be needed.

I read the code, and I think I understand what it is trying to do. It has this comment:

/* Intermediate child to prevent zombie processes.  */

As I recall, there is a "rule" on *nix that a parent must wait on a child process (or ignore SIGCHLD), or the child will be kept around in the process tables (as a "zombie"). This code seems to not want to wait for the child, and as a library doesn't want to mess with global state like signal handlers. The intermediate process exits as soon as it forks the child that will exec, so is absolved from having to wait for the child, and the parent process waits on the intermediate process allowing it to be reaped.

@jeremyd2019 I believe you’re 100% correct, and TIL: http://stackoverflow.com/questions/10932592/why-fork-twice/16655124#16655124

And I think it’s exactly the reason why it un-hangs if I manually kill the “right” pacman process (the intermediate child apparently). Which probably means that it’s actually the intermediate child that hangs (maybe because the grandchild exits too soon?).

Hope this information helps in diagnosing the root cause 🤞

Another wrinkle is that Windows doesn't have a concept of exec like on posix/unix, so Cygwin has to do some funky tricks to spawn a new process but make the posix APIs it provides act as though it is still the same process that had called exec. My personal inkling is that there is a race between that and the teardown of the intermediate child, which is what the execed process is trying to make its parent at the very same time that process is trying to exit.

Well that didn't take long:

#include <stdio.h>
#include <sys/wait.h>
#include <unistd.h>

#ifndef BINARY
#define BINARY "/bin/sleep"
#endif

#ifndef ARG
#define ARG "0.1"
#endif

int main(int argc, char ** argv)
{
    while (1)
    {
        int pid;
        printf("Starting group of 100x " BINARY " " ARG "\n");
        for (int i = 0; i < 100; ++i)
        {
            pid = fork();
            if (pid == -1)
            {
                perror("fork error");
                return 1;
            }
            else if (pid == 0)
            {
                if ((pid = fork()) == 0)
                {
                    char * const args[] = {BINARY, ARG, NULL};
                    execv(BINARY, args);
                    perror("execv failed");
                    _exit(5);
                }
                if (pid == -1)
                    _exit(1);
                else
                    _exit(0);
            }
            else
            {
                int status;
                if (waitpid(pid, &status, 0) == -1)
                {
                    perror("waitpid error");
                    return 2;
                }
                else if (status != 0)
                {
                    fprintf(stderr, "subprocess exited non-zero: %d\n", status);
                    return WEXITSTATUS(status);
                }
            }
        }
    }
    return 0;
}

built that with gcc -DBINARY=\"/bin/true\" -ggdb -o testfork testfork.c and it hung up after 11 lines of the "group of 100x" output.

On the raspberry pi/windows 10 at least, this seems to hang pretty reliably after 11 lines, with either /bin/true or /bin/sleep (leaving 0.1 as the arg). I still can't get gdb to do anything useful, I'm thinking set detach-on-fork off (or follow-fork-mode) doesn't work on cygwin.

@jeremyd2019! This is fantastic news! Excellent work. How about contributing this reproducer to the Cygwin (or cygwin-developers) mailing list?

To the surprise of probably nobody, in my experiments this reproducer does not reproduce on my x86_64 machine, neither with MSYS2 nor with Cygwin.

@jeremyd2019! This is fantastic news! Excellent work. How about contributing this reproducer to the Cygwin (or cygwin-developers) mailing list?

https://cygwin.com/pipermail/cygwin-developers/2024-May/012694.html

Another wrinkle is that Windows doesn't have a concept of exec like on posix/unix, so Cygwin has to do some funky tricks to spawn a new process but make the posix APIs it provides act as though it is still the same process that had called exec. My personal inkling is that there is a race between that and the teardown of the intermediate child, which is what the execed process is trying to make its parent at the very same time that process is trying to exit.

Hmm, I was looking at the cygwin/msys2-runtime code for something else, and came across this comment: FIXME: Is there a race here if we run this while another thread is attempting to exec()?

Don't know, but LOL anyway

I'm really sorry to add a noobish comment, but I have been following this with interest - as I have observed that the same behaviour seems to be exhibited when working with msys2/pacman via an x64 Windows instance during a docker build (hangs with Pacman).

Given the amazing digging you've done here, and the nature of working with Windows containers/docker build it seems possible that there are similar issues with races on the process forks/zombie processes on arm64 that you folks identify here - but within a Docker environment instead. I have seen it during a docker build on Windows Server Core 2022.

The workaround in my case is just to tell mys2-runtime and pacman not to auto-update/run on an ridk install: https://github.com/gocd-contrib/gocd-oss-cookbooks/blob/00c499cfffe6ae3daf0e625a1a61140b18a11a6a/provision/provision-install-packages.ps1#L60-L69

Thought I might add that in case you're looking to find a case where even x64 can exhibit similar problems. I've no idea if https://github.com/msys2/MSYS2-packages/pull/4583 would also address that, but that it might be worth adding to the mix. If it's unnecessary FUD, please ignore me :-)

@chadlwilson I guess one way to find out is see if the reproducer produces similar results in container?

@chadlwilson I guess one way to find out is see if the reproducer produces similar results in container?

Yeah, fair comment. Unfortunately I don't have a super easy way to do so as don't have an x64 machine so have to do so via GHA or similar which I haven't got around to yet.

The new 24h2 version of Windows 11 introduces a new x64 emulation engine for arm64 called Prism. I wonder if that would solve any of the hangs we've been seeing on x64-emulated processes like pacman and other tools...

Note though that Prism isn’t really “new” but it does drop some old ARMv8.0 support code, which might indeed have had an impact I guess. Also, @dscho has mentioned before that it might have been fixed in later builds of 24H2, though it might not be related to Prism (since what’s marketed now as “Prism” presumably appeared in much earlier builds of Germanium).

The new 24h2 version of Windows 11 introduces a new x64 emulation engine for arm64 called Prism. I wonder if that would solve any of the hangs we've been seeing on x64-emulated processes like pacman and other tools...

I bring bad news 😞 . My Surface Laptop Copilot+ PC arrived yesterday, and that comes with Windows 11 24H2. Today I started local dev stuff with MSYS2 and I already saw the pacman freeze twice.