[Feature] SkyWalking Python agent auto-startup in prefork servers

[Superskyyy]

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Description

When we face a prefork server like Gunicorn, we will need to manually pass a postfork snippet (example) to spin up the agent in the forked process. Not every user is aware of this, which generally causes confusion. Fortunately, this can be automatically done starting Python 3.7 thanks to this functionality.

For newcomers, this is a good first issue to start with. This is a simple yet very critical user experience enhancement feature.

Use case

Python agent should startup normally and automatically under prefork servers.

Related issues

No response

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[nekozou]

😁

[wu-sheng]

Is anything happening here?

[Superskyyy]

Is anything happening here?

My bad, I spotted some major problems with the agent startup after process fork, but previous comment was wrong so I deleted it, I will update later when I figure it out. Also removed the good first issue label since it turns out to be compilcated.

[jiang1997]

Use case

https://skywalking.apache.org/docs/skywalking-python/next/en/setup/faq/how-to-use-with-uwsgi/ But start the application by cli as showd below

sw-python run uwsgi --die-on-term \
    --http 0.0.0.0:5000 \
    --http-manage-expect \
    --master --workers 3 \
    --enable-threads \
    --threads 3 \
    --manage-script-name \
    --mount /=main:app

Potential cause

I've done a small amount of research, and this might be caused by the inherent property of Linux's fork. In Python Agent, we use a global variable _started to prevent redundant initialization.

In preforking mode of uWsgi,

1. At first, uWsgi spawns a master process in which the Agent will be initialized for the first time and _started will be set as True.
2. Then the master process spawns n worker processes in which _started remains True. So the necessary reporting threads will not be created here and nothing will be sent to oap.

Commenting this line preventing initialization inside the master process, then each worker processes will have its own Agent initialized successfully.

[Superskyyy]

Use case

skywalking.apache.org/docs/skywalking-python/next/en/setup/faq/how-to-use-with-uwsgi But start the application by cli as showd below

sw-python uwsgi --die-on-term \
    --http 0.0.0.0:5000 \
    --http-manage-expect \
    --master --workers 3 \
    --enable-threads \
    --threads 3 \
    --manage-script-name \
    --mount /=main:app

Potential cause

I've done a small amount of research, and this might be caused by the inherent property of Linux's fork. In Python Agent, we use a global variable _started to prevent redundant initialization.

In preforking mode of uWsgi,

1. At first, uWsgi spawns a master process in which the Agent will be initialized for the first time and _started will be set as True.
2. Then the master process spawns n worker processes in which _started remains True. So the necessary reporting threads will not be created here and nothing will be sent to oap.

Commenting this line preventing initialization inside the master process, then each worker processes will have its own Agent initialized successfully.

Thank you, I see where the problem is, maybe adding PID info to the __started flag so upon fork we can allow a safe restart.