BREAKING - BepInEx update requires container GLIBC update

[krstcs]

BepInEx was updated this morning and the update needs a newer version of GLIBC than what is on the container.

[MrSmiley801]

I can confirm the same thing happening

DEBUG - [351] - Server config is name: *****, port: 2456/udp, world: ****, public: 0, mod: BepInEx
DEBUG - [351] - Valheim server with PID 356 stopped
/opt/valheim/bepinex/valheim_server.x86_64: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /opt/valheim/bepinex/doorstop_libs/libdoorstop_x64.so)
/opt/valheim/bepinex/valheim_server.x86_64: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by /opt/valheim/bepinex/doorstop_libs/libdoorstop_x64.so)
/opt/valheim/bepinex/valheim_server.x86_64: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /opt/valheim/bepinex/doorstop_libs/libdoorstop_x64.so)

Version in the container (ldd --version): ldd (Debian GLIBC 2.31-13+deb11u11) 2.31

I think since this container is based on Debian 11, the fix would require a newer Debian like 12 or 13? Or another distro?

[Gyrofalcon]

I see the same stuff in my server log. Worked just fine 8 hours ago.

[Anton-To]

Same here, waiting for a fix

[lig76]

The same issue for me. Waiting for fix.

[PathFinderq]

Same issue for me too

[lig76]

I tried to downgrade BepInEx version, but the Docker image has auto-updater built-in so it didn't help.

[krstcs]

Yeah, the docker image needs to be updated, until that happens it won't work.

[JustinTArthur]

It's the libdoorstop build included in the mod pack that is built against glibc 1.38 or higher. Even if this image is updated from Debian oldstable to stable, it won't be new enough.

For those needing a workaround, here's a quick environment variable hack to freeze to the prior pack version. If you're using docker or podman, add this to your run arguments (including slashes, quotation marks, and commas): -e PRE_UPDATE_CHECK_HOOK="sed -i -e s,Valheim/\",Valheim/5.4.2202/\", -e s,\.latest\.download_url,\.download_url, /usr/local/bin/bepinex-updater"

[JustinTArthur]

If you'd prefer not to use the workaround, this is tracked upstream in NeighTools/UnityDoorstop#75

[arrowmaster]

In addition to upgrading to a new base image (it should only need to be to Debian 12, another release is on the way to lower the requirement), changes to the environment variables also need to be made for the new version of UnityDoorstop.

https://github.com/lloesche/valheim-server-docker/blob/951992b9350f53b407c3e385f6e849a26402442f/valheim-server#L36-L37 Will need to be changed to the following.

export DOORSTOP_ENABLED=1
export DOORSTOP_TARGET_ASSEMBLY="./BepInEx/core/BepInEx.Preloader.dll"

Or it will probably work to set all four but I've not tested it on either the new or old version.

[proofquantu]

I managed to build the container using debian:bookworm-slim (aka debian12) added the change to valheim-server file as per @arrowmaster fine suggestion. Was getting an error in the docker compose logs "unable to load shared object libdoorstop_x64.so" because there was no library with that name in doorstop_libs ! Created a symlink libdoorstop_x64.so -> libdoorstopx64.so in $HOME/valheim-server/data/bepinex/doorstop_libs

Valheim server starts up - however...

Still can't join the server because other required mods don't load eg. valheimplus and Jotunn based mods ie xportal etc. So it looks like downstream mods will also need to be updated to work with this new version of BepInEx.

[Santya95]

It's the libdoorstop build included in the mod pack that is built against glibc 1.38 or higher. Even if this image is updated from Debian oldstable to stable, it won't be new enough.

For those needing a workaround, here's a quick environment variable hack to freeze to the prior pack version. If you're using docker or podman, add this to your run arguments (including slashes, quotation marks, and commas): -e PRE_UPDATE_CHECK_HOOK="sed -i s,Valheim/\",Valheim/5.4.2202/\", /usr/local/bin/bepinex-updater"

This workaround works flawlessly thanks for sharing, it's just Jotunn not loading server side, do you have any suggestion?

[JustinTArthur]

This workaround works flawlessly thanks for sharing, it's just Jotunn not loading server side, do you have any suggestion?

Thanks, it may not have been flawless afterall. I've updated the workaround comment to fix the download URL for zip extraction.

The hook env var has gotten bigger/messier, so hopefully this can be fixed image-side soon :)

[lig76]

So what now ? The only reasonable approach to have mods working permanently is next update of BepInEx built with older glibc ?

[JustinTArthur]

So what now ? The only reasonable approach to have mods working permanently is next update of BepInEx built with older glibc ?

@ghorsington's BepInEx pack on Thunderstore (which this image downloads) now has libdoorstop built for a more accessible glibc 2.35+. As for what's next, basically two things:

• In the short term, this image's build can be updated to use a newer base Debian image and to update the libdoorstop env vars for the new scheme.
• In the long term, a strategy could be developed to prevent future breakage from the auto-update process, prevent surprise via version freezing, or allow opt-out of the BepInEx pack auto-updates. @lloesche may have some ideas of where he'd like to take the process.

[krstcs]

While I understand why the game server containers tend to do internal update processes (it's easier for the service to check for updates than for the maintainer to have a separate service that just does it in order to start a new image build), containers shouldn't really do that. They should be rebuilt if the server version needs to be updated. That would make the image easier to maintain and stop some of the "auto-update into an error" that we got here.

[krstcs]

Of course it would also be nice if these game devs would release an official container version of their servers and keep it updated for us.

[Mjs0333]

I'm having difficulty implementing JustinTArthur's workaround. Where specifically should these arguments be added if using docker compose? If I modify valheim-server/data/server/docker_start_server.sh and add in the extra arguments to the docker run line and start the server, valheim-updater appears to download a different BepInEx version but that also results in the original pre-edited docker_start_server.sh getting pulled down and the server still fails to find glibc afterwards.

Is there somewhere else that I need to feed these arguments?

[JustinTArthur]

I'm having difficulty implementing JustinTArthur's workaround. Where specifically should these arguments be added if using docker compose?

My workaround was for docker run and podman run on a shell like bash or cmd.exe

For docker compose, it supports two YAML syntaxes for supplying environment variables and I'll make a shortened version of what it looks like in either:

services:
  valheim:
    environment:
      - PRE_UPDATE_CHECK_HOOK=sed -i -e s,Valheim/",Valheim/5.4.2202/", -e s,\.latest\.download_url,\.download_url, /usr/local/bin/bepinex-updater

services:
  valheim:
    environment:
      PRE_UPDATE_CHECK_HOOK: sed -i -e s,Valheim/",Valheim/5.4.2202/", -e s,\.latest\.download_url,\.download_url, /usr/local/bin/bepinex-updater

You likely have the rest of your image config env vars set in that same region.

[Mjs0333]

@JustinTArthur Thank you very much for your quick response and clarifying for a newbie! Adding in your suggestion into my existing YAML file worked like a charm and the wife and I can now happily play again.

[lig76]

@JustinTArthur your workaround with PRE_UPDATE_CHECK_HOOK works like a charm, thank you ! Blocking the updater was in the back of my head, but I didn't know how.

Thanks !

[JustinTArthur]

@JustinTArthur, hey man thanks again for the clarification. I used both the versions of the hook you provided, both provides me the same output looking at logs…

Looks like the double-quotes got included verbatim in the env var, which shouldn't happen when conveyed from a shell. How are you launching the container? If using docker compose see https://github.com/lloesche/valheim-server-docker/issues/752#issuecomment-3162258944

[Santya95]

@JustinTArthur, hey man thanks again for the clarification. I used both the versions of the hook you provided, both provides me the same output looking at logs…

Looks like the double-quotes got included verbatim in the env var, which shouldn't happen when conveyed from a shell. How are you launching the container? If using docker compose see #752 (comment)

I'm incomporating PRE_UPDATE_CHECK_HOOK string in a .env file and launching docker run like this:

docker run -d \
  --name valheim-server \
  --env-file /opt/valheim-server/.env \
  -v /opt/valheim-server/config:/config \
  -p 2456-2458:2456-2458/udp \
  --restart unless-stopped \
  lloesche/valheim-server

Now i discarded the env file for an explicit docker run bash command and it looks to work just fine!

[MrSmiley801]

I'm having difficulty implementing JustinTArthur's workaround. Where specifically should these arguments be added if using docker compose?

My workaround was for docker run and podman run on a shell like bash or cmd.exe

For docker compose, it supports two YAML syntaxes for supplying environment variables and I'll make a shortened version of what it looks like in either:

services: valheim: environment: - PRE_UPDATE_CHECK_HOOK=sed -i -e s,Valheim/",Valheim/5.4.2202/", -e s,.latest.download_url,.download_url, /usr/local/bin/bepinex-updater services: valheim: environment: PRE_UPDATE_CHECK_HOOK: sed -i -e s,Valheim/",Valheim/5.4.2202/", -e s,.latest.download_url,.download_url, /usr/local/bin/bepinex-updater You likely have the rest of your image config env vars set in that same region.

I added that to my compose.yaml and looks like the command works but then I run into permission issue. I'm running PUID/PGID 1000 and the folders should all have the correct permissions. I even tried a chmod -R 777 on the opt and config folders but this shows in the logs. It doesn't complete the sed command so it downloads the latest version of bepinex and still crashes.

In the logs:

Aug 7 07:51:21 supervisord: valheim-updater INFO - Running pre update check hook: sed -i -e s,Valheim/",Valheim/5.4.2202/", -e s,.latest.download_url,.download_url, /usr/local/bin/bepinex-updater Aug 7 07:51:21 supervisord: valheim-updater sed: couldn't open temporary file /usr/local/bin/sed77uKm7: Permission denied

My YAML looks like

services:
  valheim-server:
    image: ghcr.io/lloesche/valheim-server
    container_name: valheim-server
    cap_add:
      - sys_nice
    volumes:
      - ./config:/config
      - ./opt/valheim:/opt/valheim
    ports:
      - 2456-2458:2456-2458/udp
      - 9105:9105/tcp
    environment:
      - BEPINEX=true
      - PRE_UPDATE_CHECK_HOOK=sed -i -e s,Valheim/",Valheim/5.4.2202/", -e s,\.latest\.download_url,\.download_url, /usr/local/bin/bepinex-updater
      - SERVER_NAME=Private Server
      - SERVER_PORT=2456
      - WORLD_NAME=omitted
      - SERVER_PASS=omitted
      - SERVER_PUBLIC=false
      - TZ=America/New_York
      - BACKUPS=true
      - BACKUPS_MAX_AGE=2
      - STEAMCMD_ARGS=validate
      - PUBLIC_TEST=false
      - PUID=1000
      - PGID=1000
      - DNS_1=8.8.8.8 
      - DNS_2=8.8.4.4 
      - SUPERVISOR_HTTP=true
      - SUPERVISOR_HTTP_PORT=9105
      - SUPERVISOR_HTTP_USER=omitted
      - SUPERVISOR_HTTP_PASS=omitted
    restart: unless-stopped
    stop_grace_period: 2m
networks: {}

[arrowmaster]

I'm running PUID/PGID 1000 and the folders should all have the correct permissions.

@MrSmiley801 What is most likely happening is the PRE_UPDATE_CHECK_HOOK is running as PUID but /usr/local/bin/bepinex-updater is owned by root in the container. Using PRE_SUPERVISOR_HOOK looks like it will probably run as root in the container before it changes to PUID. Just try changing PRE_UPDATE_CHECK_HOOK to PRE_SUPERVISOR_HOOK in your compose.yaml.

[MrSmiley801]

@arrowmaster that did the trick! Thank you!!!!

I saw the pull request for upstream fix too (https://github.com/NeighTools/UnityDoorstop/pull/76) and appreciate all your help immensely.

[Gyrofalcon]

Just curious; Is there any downsides, if we edit the /usr/local/bin/bepinex-updater manually, and don't run the PRE_UPDATE_CHECK_HOOK argument? Other that we'll need to do it again, every time we rebuild the container/stack.

[JustinTArthur]

Just curious; Is there any downsides, if we edit the /usr/local/bin/bepinex-updater manually, and don't run the PRE_UPDATE_CHECK_HOOK argument? Other that we'll need to do it again, every time we rebuild the container/stack.

That's fine. You can avoid it resetting by just making your own copy of bepinex-updater and bind-mounting it to the container's /usr/local/bin/bepinex-updater

e.g.

docker run --volume $HOME/my_image_changes/bepinex-updater:/usr/local/bin/bepinex-updater …

[Gyrofalcon]

Just curious; Is there any downsides, if we edit the /usr/local/bin/bepinex-updater manually, and don't run the PRE_UPDATE_CHECK_HOOK argument? Other that we'll need to do it again, every time we rebuild the container/stack.

That's fine. You can avoid it resetting by just making your own copy of bepinex-updater and bind-mounting it to the container's /usr/local/bin/bepinex-updater

e.g.

docker run --volume $HOME/my_image_changes/bepinex-updater:/usr/local/bin/bepinex-updater …

Thank you.

[Delvjn]

I can't get the workaround to function anymore. It worked for a little while but now i get these errors:

Aug 12 14:45:11 supervisord: valheim-updater DEBUG - [50] - BepInEx is enabled - running updater Aug 12 14:45:11 supervisord: valheim-updater curl: (22) The requested URL returned error: 400 Aug 12 14:45:11 supervisord: valheim-updater FATAL - Error: could not retrieve BepInEx release info from Thunderstore.io API Aug 12 14:45:11 supervisord: valheim-updater ERROR - Failed to run BepInEx updater - retrying later - check your networking and volume access permissions

I use this to run the container: sudo docker run --restart unless-stopped --network host -d --name valheim-server3 --cap-add=sys_nice --stop-timeout 120 -v $HOME/valheim-server/config:/config -v $HOME/valheim-server/data:/opt/valheim -e SERVER_NAME="123" -e WORLD_NAME="123" -e SERVER_PASS="123" -e BACKUPS_MAX_COUNT=10 -e BACKUPS_MAX_AGE=100000 -e TZ="Europe/Oslo" -e BEPINEX=true -e PRE_UPDATE_CHECK_HOOK="sed -i -e s,Valheim/",Valheim/5.4.2202/", -e s,.latest.download_url,.download_url, /usr/local/bin/bepinex-updater" lloesche/valheim-server

Does anyone know how to fix it? Are there any updates on when we will get an permanent fix on this container?