Common segfault in `restore_og` when running `SWOOLE_PROCESS` server.

[cjavad]

trafficstars

Hi, over the last couple of days i've seen a huge influx in worker processes that dies due to the same segmentation fault. This has sadly only been happening in our production systems and not in testing yet, but i've managed to capture 2 core dumps over 1 hour using swoole compiled with debug symbols. We are running swoole 6.0.2 with php 8.4.7 on linux 6.8.0-58-generic under docker in alpine 3.21, and i've provided a Dockerfile environment that matches the exact environment of the application.

Here is an except of the issue:

It appears that the PHPContext output_ptr gets corrupted somehow, but i am not able to run more intrusive debugging tools such as xdebug or valgrind since its production, but any ideas are welcome. This happens consistently - we are not using any native php modules that should impact this and have in periods experienced up to 20 of these faults in a single day.

To interactively inspect the core dumps use the following docker file as a base environment:

FROM php:8.4-cli-alpine AS build

# Install system dependencies
RUN apk --no-cache add \
    postgresql-dev \
    libzip-dev \
    autoconf \
    gcc \
    g++ \
    make \
    curl \
    libpng-dev \
    libjpeg-turbo-dev \
    freetype-dev \
    libxml2-dev \
    openssl-dev \
    curl-dev \
    linux-headers \
    sqlite-dev

RUN docker-php-ext-configure gd --with-freetype --with-jpeg \
    && docker-php-ext-install curl pdo_mysql pdo_sqlite gd sockets soap zip bcmath \
    && pecl install redis \
    && docker-php-ext-enable redis opcache \
    && pecl install --configureoptions 'enable-sockets="yes" enable-openssl="yes" enable-mysqlnd="yes" enable-swoole-curl="yes" enable-cares="no" enable-brotli="no" enable-zstd="no" enable-swoole-pgsql="no" with-swoole-odbc="no" with-swoole-oracle="no" enable-swoole-sqlite="no" enable-swoole-thread="no" enable-iouring="no" enable-debug="yes"' swoole-6.0.2 \
    && docker-php-ext-enable swoole


FROM php:8.4-cli-alpine

RUN apk --no-cache add \
    libzip \
    libpng \
    libjpeg-turbo \
    freetype \
    libxml2 \
    openssl \
    curl \
    libstdc++ \
    sqlite \
    gdb \
    binutils

COPY --from=build /usr/local/etc/php/conf.d /usr/local/etc/php/conf.d 
COPY --from=build /usr/local/lib/php/extensions /usr/local/lib/php/extensions

With GDB inside this container open the core dump with gdb -c [COREDUMPFILE] and use info sharedlibrary to get the start offset of swoole.so (will be something like 0x000073bf19013900).

With this offset load the provided 20240924-debug-swoole.so file with add-symbol-file 20240924-debug-swoole.so [OFFSET] and choose y to read the symbol file. You now have the correct debug symbols to inspect the call-stack etc.

A quick set of commands to get testing fast:

mkdir -p /tmp/dumps
cd /tmp/dumps
curl -O https://my-files.javad.sh/dumps/20240924-debug-swoole.so.gz
curl -O [COREDUMPFILE1]
curl -O [COREDUMPFILE2]
gunzip *
nano Dockerfile # Write dockerfile
docker build . -t gdb-test 
docker run --rm -it -u root -v /tmp/dumps:/tmp/dumps --entrypoint ash gdb-test
gdb -c /tmp/dumps/[COREDUMPFILE1]
> info sharedlibrary
> add-symbol-file /tmp/dumps/20240924-debug-swoole.so [OFFSET]
> bt

Since all the coredumps are from the same process (including the 10 new ones i got today) the offset is always 0x000073bf19013900 which can be used to skip docker and just use it directly.

I am hosting the associated .so and coredump files.

• 20240924-debug-swoole.so: https://my-files.javad.sh/dumps/20240924-debug-swoole.so.gz

The core dump files have been redacted for all sensitive environment variables etc. but i would prefer sharing them privately, or with GPG, for convenience i've GPG'ed them for the swoole maintainers with public GPG keys, but please provide an email address or a GPG-key you'd like to receive them with.

info.gpg.txt

[matyhtf]

Please try to trace it with Valrind

USE_ZEND_ALLOC=0 valgrind php your_code.php

[cjavad]

Please try to trace it with Valrind

USE_ZEND_ALLOC=0 valgrind php your_code.php

Yes i would love to be able to do this, this only happens in production though, i am thinking of doing a custom build with ultra-specific logging to minimize performance impact, if you had any specific things you wanted to track or enable i could do that.

At a high level it appears that the origin_ctx is invalid (maybe double free) when destroy_context is called on a child context. If you wanted some additional information i've emailed you the files as well.

I'll try to create a minimal repo of this issue, do you know of any way we could see what PHP functions the coroutines are based on, that might help me be able to do that.

On a side note i am running all our test instances with valgrind as you suggested above, i'll see what i can do to replicate the issue.

[cjavad]

I have been able to confirm the issue to be an use after free from Coroutine::on_close -> PHPContext::on_close -> efree(ctx) being called before restore_og on the PHPContext perhaps a race condition of sorts.

I did this by explicitly overwriting the struct with 0xAA bytes before freeing it so the execution path is more or less correct, see https://github.com/SimplyPrint/swoole-src/compare/d5cd7935442a62cd6856ab58029889b8d15efbd3..v6.0.2-output-ptr-track.

I am still open to ideas on how to trace this in a minimal invasive way as the issue still only has occurred in our production systems.

Edit:

Some more context, most likely the issue occurs then a coroutine (A) points to an origin (B) that has been closed with Coroutine::close from check_end. When A exists out from main_func it reschedules the origin triggering the fault. Inspecting more coredumps it appears that A->co->origin (get origin) is 0x0 so it returns main_context which somehow is invalid? But i am not able to inspect the .bss regions in my coredumps so i cannot see the values of Swoole::coroutines nor main_context.

[matyhtf]

@cjavad You can run your program with asan to reproduce the issue and trace the cause of the crash.

https://github.com/swoole/swoole-src/blob/master/docs/ISSUE.md#address-sanitizer https://github.com/swoole/swoole-src/blob/master/Dockerfile

[cjavad]

Hi @matyhtf, still no luck reproducing in testing, i'll have to evaluate if the 2x performance degradation makes sense to try to run it in production.

At a high level this issue is occurring about 3-6 times every day, but with relatively low impact now that the other bug that was triggered by worker crashes has been solved. From our side we wont spend too much time on this issue and i would be fine to close this ticket for now. In the future we will probably move away from swoole to resolve this issue for good.

[matyhtf]

@cjavad Thank you for your feedback on the issue. This issue is somewhat complex, and we did not find any exceptions in the test cases where the output buffer was switched in the coroutine. In theory, it will restore the OG global variables during coroutine switching and destruction. PHP code cannot break the underlying memory structure of OG unless there are other extensions that steal this pointer. Please use asan as much as possible to track this issue and help us find the root cause of the memory exception.

[cjavad]

Hi again @matyhtf recently getting very consistent segfaults, all in php unserialize. I am noticing swoole does not seem to handle the BG(v) global variables used by var_serializer.c which could be a root cause for this issue, does that ring a bell?

This error was fixed by switching to igbinary serializer, so i guess there is some reentrancy issues in the serialize + swoole code, but the original segfault persists.

Some more context:

The segfault frequency absolutely dropped (from multiple times an hour) to a couple of times a day, by disabling task workers in our code (we didn't need them and could just spawn more coroutines), so the serialization aspect of these errors seems more likely with that evidence.

I will try to attempt to replicate this workflow (workers, task workers, redis, doctrine) all using serialize/unserialize and see if i can replicate the error independently.

[matyhtf]

@cjavad This needs to be tested. If __sleep/__wakeup is used and the coroutine is suspended in these functions, and multiple coroutines are var_serializer at the same time, the wrong memory may be accessed.

The context state must be saved when the coroutine is suspended. Unfortunately, the underlying code of PHP zendvm uses many global variables, and swoole cannot cover all cases.

[matyhtf]

https://github.com/swoole/swoole-src/commit/946039fd9046e426e18e830081ad4fc51b846da3

Now, when switching coroutines, global variables related to serialization and deserialization will be saved.

[cjavad]

Will test, thank you!

Edit: Deployed into production successfully, no immediately faults, i'll report back tomorrow, and then again after a week, fingers crossed haha.

The current pattern of the segfault is that it about once a day likes to appear about 10 times in a quick succession, then not appear again. It does not overlap with our most active hours or anything else so pretty odd.

[cjavad]

That did something, getting a rare segfault in swoole::MsgQueue::MsgQueue(int, bool, int) + 48 in section .text now, these segfaults have the same very consistent offset so i doubt there is any nondeterministic memory behavior.

[NathanFreeman]

Can you please try again? We just modified some of the logic in this part. If the issue still occurs, see if we can use valgrind to check where the problem lies.

[cjavad]

Had 6.1.0 running in production for a couple of days, the fault still appeared in its normal frequency sadly, but i also had to rollback to 6.0.2 due to a recurring issue with workers not accepting connections, will look into it more. If you had some resources and were very interested in this problem we are more than happy to share some more details privately, but from our side i might try to do an isolated environment simulating the load to see if i can trigger the errors consistently.