palworld-server-docker icon indicating copy to clipboard operation
palworld-server-docker copied to clipboard

Published 20 hours ago • verified publisher icon thijsvanloef

Use Supervisord to manage processes

Open Luatan opened this issue 1 year ago • 3 comments

Context

  • This will enable us to start, stop, restart different processes
  • No Container reboot is needed to restart the main Palworld process
  • Enables updates, restores and other scripts can be run within the container without restarting it.
  • closes #399

Choices

  • Using Supervisord to manage the processes.
  • Adds Triggers for Discord Notifications tied to the application state of the palworld server
  • Adds a separate script for starting the server, to only run start.sh once on container start.
  • Update and reboot scripts use supervisord to shutdown and restart the server
  • Restore Script uses supervisord to shutdown and restart the server

Processes Managed by Supervisord

  • Palworld
  • Palworld-Event-Handler
  • Supercronic

Test instructions

Environment Variables

These Environment variables can be used for all test cases

DISCORD_WEBHOOK_URL: "your test webhook"
UPDATE_ON_BOOT: false
BACKUP_ENABLED: false
PLAYER_LOGGING_POLL_PERIOD: 10

Always Running Services

These services should always be running if enabled in the config

  • Palworld-Server
  • Crontab
  • Player Logging (can be disabled)

Test

  1. Start Container
  2. Verify that the Server logs correctly to docker logs
  3. Verify that the discord notifications are being sent correctly
  4. Check if all mentioned above are processes are running docker exec -it palworld-server supervisorctl status
  5. Join the server with an palworld client
  6. Verify that the player_logging scripts sends a message as broadcast and in the logs
  7. Stop the container
  8. Verify with docker logs in another terminal that the server stops correctly
  9. Verify that the discord notifications are being sent correctly

Scripts

These scripts use supervisord to shutdown the server if necessary.

  • Restore
  • Update
  • Auto Reboot

Restore Test

  1. Start the container
  2. Create a backup
  3. restore the backup with the command docker exec -it palworld-server restore
  4. Check the logs and verify that the server shutdown and the restore is successfull

Update Test

  1. Start the container
  2. Run the command docker exec palworld-server update and verify that the server is up to date
  3. Change the "manifest" value in palworld\steamapps\appmanifest_2394010.acf on Line 31 to a different value
  4. Run the command docker exec palworld-server update and verify that an update is found
  5. Verify that the server is being shutdown without stopping the container
  6. Verify that the server is started again after steam has completed the "update"

Auto Reboot Test

  1. Start the container
  2. Run the command docker exec palworld-server auto_reboot.sh
  3. Verify that the server shutsdown without stopping the container
  4. Verify that the server starts again

Checklist before requesting a review

  • [ ] I have performed a self-review of my code
  • [ ] I've added documentation about this change to the README.
  • [ ] I've not introduced breaking changes.

Luatan avatar Feb 17 '24 15:02 Luatan

What's left to do on this? @Luatan

xHyperElectric avatar Mar 05 '24 15:03 xHyperElectric

What's left to do on this? @Luatan

I didn't really had the time to continue with this. But I am currently struggling with the eventlistener. Since I have added another process (the player logging) the behavior is really strange. If the player logging is enabled it does not send the same events

to see it you can add the line

LogWarn "$header" >&2

to the palworld_state_handler.sh before the if statement which checks the service name

then if I enable the player logging in the docker compose file I get this output:

palworld-server  | Compiling PalWorldSettings.ini done!
palworld-server  | ****GENERATING CRONTAB****
palworld-server  | Player Logging enabled
palworld-server  | time="2024-03-05T19:52:27Z" level=info msg="read crontab: /home/steam/server/crontab"
palworld-server  | ver:3.0 server:supervisor serial:0 pool:palworld-events poolserial:0 eventname:PROCESS_STATE_STARTING len:80
palworld-server  | processname:palworld-events groupname:palworld-events from_state:STOPPED tries:0ver:3.0 server:supervisor serial:1 pool:palworld-events poolserial:1 eventname:PROCESS_STATE_STARTING len:64
palworld-server  | processname:crontab groupname:crontab from_state:STOPPED tries:0ver:3.0 server:supervisor serial:2 pool:palworld-events poolserial:2 eventname:PROCESS_STATE_STARTING len:66
palworld-server  | processname:palworld groupname:palworld from_state:STOPPED tries:0ver:3.0 server:supervisor serial:3 pool:palworld-events poolserial:3 eventname:PROCESS_STATE_STARTING len:78
palworld-server  | *********STARTING SERVER*********
palworld-server  | processname:player_logging groupname:player_logging from_state:STOPPED tries:0ver:3.0 server:supervisor serial:4 pool:palworld-events poolserial:4 eventname:PROCESS_STATE_RUNNING len:80
palworld-server  | processname:palworld-events groupname:palworld-events from_state:STARTING pid:36ver:3.0 server:supervisor serial:5 pool:palworld-events poolserial:5 eventname:PROCESS_STATE_RUNNING len:78
palworld-server  | processname:player_logging groupname:player_logging from_state:STARTING pid:39ver:3.0 server:supervisor serial:6 pool:palworld-events poolserial:6 eventname:PROCESS_STATE_RUNNING len:64
palworld-server  | [S_API] SteamAPI_Init(): Loaded local 'steamclient.so' OK.
palworld-server  | Shutdown handler: initalize.
palworld-server  | - Existing per-process limit (soft=1048576, hard=1048576) is enough for us (need only 1048576)
palworld-server  | Increasing per-process limit of core file size to infinity.
palworld-server  | CAppInfoCacheReadFromDiskThread took 0 milliseconds to initialize
palworld-server  | Setting breakpad minidump AppID = 2394010
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamUser021 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamFriends017 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface STEAMAPPS_INTERFACE_VERSION008 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamNetworkingUtils004 before SteamAPI_Init succeeded.
palworld-server  | Waiting for RCON port to open to show player logging...
palworld-server  | processname:crontab groupname:crontab from_state:STARTING pid:37ver:3.0 server:supervisor serial:7 pool:palworld-events poolserial:7 eventname:PROCESS_STATE_RUNNING len:66
palworld-server  | Waiting for RCON port to open to show player logging...

and if it is disabled:

palworld-server  | ****GENERATING CRONTAB****
palworld-server  | time="2024-03-05T19:55:16Z" level=info msg="read crontab: /home/steam/server/crontab"
palworld-server  | ver:3.0 server:supervisor serial:0 pool:palworld-events poolserial:0 eventname:PROCESS_STATE_STARTING len:80
palworld-server  | processname:palworld-events groupname:palworld-events from_state:STOPPED tries:0ver:3.0 server:supervisor serial:1 pool:palworld-events poolserial:1 eventname:PROCESS_STATE_STARTING len:64
palworld-server  | processname:crontab groupname:crontab from_state:STOPPED tries:0ver:3.0 server:supervisor serial:2 pool:palworld-events poolserial:2 eventname:PROCESS_STATE_STARTING len:66
palworld-server  | processname:palworld groupname:palworld from_state:STOPPED tries:0ver:3.0 server:supervisor serial:3 pool:palworld-events poolserial:3 eventname:PROCESS_STATE_RUNNING len:80
palworld-server  | *********STARTED SERVER*********
palworld-server  | processname:palworld-events groupname:palworld-events from_state:STARTING pid:35ver:3.0 server:supervisor serial:4 pool:palworld-events poolserial:4 eventname:PROCESS_STATE_RUNNING len:64
palworld-server  | [S_API] SteamAPI_Init(): Loaded local 'steamclient.so' OK.
palworld-server  | Shutdown handler: initalize.
palworld-server  | - Existing per-process limit (soft=1048576, hard=1048576) is enough for us (need only 1048576)
palworld-server  | Increasing per-process limit of core file size to infinity.
palworld-server  | CAppInfoCacheReadFromDiskThread took 0 milliseconds to initialize
palworld-server  | Setting breakpad minidump AppID = 2394010
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamUser021 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamFriends017 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface STEAMAPPS_INTERFACE_VERSION008 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamNetworkingUtils004 before SteamAPI_Init succeeded.
palworld-server  | processname:crontab groupname:crontab from_state:STARTING pid:36ver:3.0 server:supervisor serial:5 pool:palworld-events poolserial:5 eventname:PROCESS_STATE_RUNNING len:66

But we should get both the events:

  1. Starting
  2. Started

on shutdown there is a similar issue:

  1. Stopping
  2. Stopped

I don't know what I am doing wrong. I think the easiest way is to test it with webbooks. You can see it there very clearly.

If that issue is resolved. The

  • Restore
  • Update
  • Auto Reboot scripts need to be changed to use supervisord to shutdown and start the server

Luatan avatar Mar 05 '24 20:03 Luatan

@Luatan This solved it.

diff --git a/services/listener/palworld_state_handler.sh b/services/listener/palworld_state_handler.sh
index b8b103a..c61d5bf 100644
--- a/services/listener/palworld_state_handler.sh
+++ b/services/listener/palworld_state_handler.sh
@@ -21,6 +21,12 @@ while true; do
     printf "READY\n"
     read -r header
     parse_headers "$header"
+    LogWarn "${header}" >&2
+    if [ -n "${len}" ]; then
+        read -n "${len}" data
+        LogWarn "${data}" >&2
+        parse_headers "${data}"
+    fi
 
     if [ "${processname}" = "palworld" ]; then
         case $eventname in

ENABLE_PLAYER_LOGGING=true

palworld-server  | ****EXECUTING USERMOD****
palworld-server  | usermod: no changes
palworld-server  | ****Checking for new update****
palworld-server  | Current Version: 3872500952532478729
palworld-server  | The Server is up to date!
palworld-server  | ****GENERATING CONFIG****
palworld-server  | Using Env vars to create PalWorldSettings.ini
palworld-server  | ****Compiling PalWorldSettings.ini****
palworld-server  | Compiling PalWorldSettings.ini done!
palworld-server  | ****GENERATING CRONTAB****
palworld-server  | BACKUP_ENABLED=true
palworld-server  | Adding cronjob for auto backups
palworld-server  | Player Logging enabled
palworld-server  | time="2024-05-20T02:32:37Z" level=info msg="read crontab: /home/steam/server/crontab"
palworld-server  | The file already exists: /palworld/Pal/Binaries/Linux/steamclient.so
palworld-server  | ver:3.0 server:supervisor serial:0 pool:palworld-events poolserial:0 eventname:PROCESS_STATE_STARTING len:80
palworld-server  | processname:palworld-events groupname:palworld-events from_state:STOPPED tries:0
palworld-server  | ver:3.0 server:supervisor serial:1 pool:palworld-events poolserial:1 eventname:PROCESS_STATE_STARTING len:64
palworld-server  | processname:crontab groupname:crontab from_state:STOPPED tries:0
palworld-server  | ver:3.0 server:supervisor serial:2 pool:palworld-events poolserial:2 eventname:PROCESS_STATE_STARTING len:66
palworld-server  | processname:palworld groupname:palworld from_state:STOPPED tries:0
palworld-server  | *********STARTING SERVER*********
palworld-server  | ver:3.0 server:supervisor serial:3 pool:palworld-events poolserial:3 eventname:PROCESS_STATE_STARTING len:78
palworld-server  | processname:player_logging groupname:player_logging from_state:STOPPED tries:0
palworld-server  | ver:3.0 server:supervisor serial:4 pool:palworld-events poolserial:4 eventname:PROCESS_STATE_RUNNING len:80
palworld-server  | processname:palworld-events groupname:palworld-events from_state:STARTING pid:51
palworld-server  | ver:3.0 server:supervisor serial:5 pool:palworld-events poolserial:5 eventname:PROCESS_STATE_RUNNING len:78
palworld-server  | processname:player_logging groupname:player_logging from_state:STARTING pid:54
palworld-server  | [S_API] SteamAPI_Init(): Loaded local 'steamclient.so' OK.
palworld-server  | CAppInfoCacheReadFromDiskThread took 0 milliseconds to initialize
palworld-server  | Setting breakpad minidump AppID = 2394010
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamUser021 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamFriends017 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface STEAMAPPS_INTERFACE_VERSION008 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamNetworkingUtils004 before SteamAPI_Init succeeded.
palworld-server  | ver:3.0 server:supervisor serial:6 pool:palworld-events poolserial:6 eventname:PROCESS_STATE_RUNNING len:64
palworld-server  | processname:crontab groupname:crontab from_state:STARTING pid:52
palworld-server  | Waiting for RCON port to open to show player logging...
palworld-server  | ver:3.0 server:supervisor serial:7 pool:palworld-events poolserial:7 eventname:PROCESS_STATE_RUNNING len:66
palworld-server  | processname:palworld groupname:palworld from_state:STARTING pid:53
palworld-server  | *********STARTED SERVER*********

ENABLE_PLAYER_LOGGING=false

palworld-server  | ****EXECUTING USERMOD****
palworld-server  | usermod: no changes
palworld-server  | ****Checking for new update****
palworld-server  | Current Version: 3872500952532478729
palworld-server  | The Server is up to date!
palworld-server  | ****GENERATING CONFIG****
palworld-server  | Using Env vars to create PalWorldSettings.ini
palworld-server  | ****Compiling PalWorldSettings.ini****
palworld-server  | Compiling PalWorldSettings.ini done!
palworld-server  | ****GENERATING CRONTAB****
palworld-server  | BACKUP_ENABLED=true
palworld-server  | Adding cronjob for auto backups
palworld-server  | time="2024-05-20T02:34:56Z" level=info msg="read crontab: /home/steam/server/crontab"
palworld-server  | The file already exists: /palworld/Pal/Binaries/Linux/steamclient.so
palworld-server  | ver:3.0 server:supervisor serial:0 pool:palworld-events poolserial:0 eventname:PROCESS_STATE_STARTING len:80
palworld-server  | processname:palworld-events groupname:palworld-events from_state:STOPPED tries:0
palworld-server  | ver:3.0 server:supervisor serial:1 pool:palworld-events poolserial:1 eventname:PROCESS_STATE_STARTING len:64
palworld-server  | processname:crontab groupname:crontab from_state:STOPPED tries:0
palworld-server  | ver:3.0 server:supervisor serial:2 pool:palworld-events poolserial:2 eventname:PROCESS_STATE_STARTING len:66
palworld-server  | processname:palworld groupname:palworld from_state:STOPPED tries:0
palworld-server  | *********STARTING SERVER*********
palworld-server  | ver:3.0 server:supervisor serial:3 pool:palworld-events poolserial:3 eventname:PROCESS_STATE_RUNNING len:80
palworld-server  | processname:palworld-events groupname:palworld-events from_state:STARTING pid:50
palworld-server  | [S_API] SteamAPI_Init(): Loaded local 'steamclient.so' OK.
palworld-server  | CAppInfoCacheReadFromDiskThread took 0 milliseconds to initialize
palworld-server  | Setting breakpad minidump AppID = 2394010
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamUser021 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamFriends017 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface STEAMAPPS_INTERFACE_VERSION008 before SteamAPI_Init succeeded.
palworld-server  | [S_API FAIL] Tried to access Steam interface SteamNetworkingUtils004 before SteamAPI_Init succeeded.
palworld-server  | ver:3.0 server:supervisor serial:4 pool:palworld-events poolserial:4 eventname:PROCESS_STATE_RUNNING len:64
palworld-server  | processname:crontab groupname:crontab from_state:STARTING pid:51
palworld-server  | ver:3.0 server:supervisor serial:5 pool:palworld-events poolserial:5 eventname:PROCESS_STATE_RUNNING len:66
palworld-server  | processname:palworld groupname:palworld from_state:STARTING pid:52
palworld-server  | *********STARTED SERVER*********

MusclePr avatar May 20 '24 02:05 MusclePr

Hey @MusclePr, thanks for helping. Still need to test it, but I was wondering if this feature is something that should still be implemented. I haven't had the time to really follow the progress of the container. And would like to get some feedback first, before I go ahead and invest more time into this. Would you mind giving your take about this @thijsvanloef .

Luatan avatar May 27 '24 16:05 Luatan

Hi @Luatan, I am still not really sure, I'll think about it and get back to you, sorry I was not able to check this out sooner.

thijsvanloef avatar May 29 '24 13:05 thijsvanloef

Deciding to close this issue. Supervisored is not the way I want to move forward with managing processes. Instead opening a feature request to handle the processes with s6 overlay

thijsvanloef avatar Jun 14 '24 11:06 thijsvanloef

Deciding to close this issue. Supervisored is not the way I want to move forward with managing processes. Instead opening a feature request to handle the processes with s6 overlay

I was actually also looking for alternatives, but couldn't find any. No clue how I didn't find s6 overlay. it looks promising as far as I have seen. 👍

Luatan avatar Jun 14 '24 21:06 Luatan