self-host icon indicating copy to clipboard operation
self-host copied to clipboard
verified publisher icon bitwarden

admin, api, identity core dump after 2025.2.0-beta

Open ahayzen opened this issue 11 months ago • 1 comments

Self-host installation scripts issue

  • [x] I understand this repository only tracks self-host installation scripts and any other issues should be reported in bitwarden/server

Steps To Reproduce

If you run the following command / start a 2025.3.0-beta container it will fail to start. (yes i know i don't have any volumes this is an example to show it happens with an empty environment, and this also happens with podman or docker).

$ podman run -it --rm \
  -e BW_INSTALLATION_ID=00000000-0000-0000-0000-000000000000 \
  -e BW_INSTALLATION_KEY=xxxxxxxxxxxx \
  -e BW_DOMAIN=test.local \
  -e BW_DB_PROVIDER=sqlite \
  -e globalSettings__disableUserRegistration=true \
  -p 8080 \
  ghcr.io/bitwarden/self-host:2025.3.0-beta@sha256:a7da29e3eabfd17c727971b6916a1651c30e50c3432fb195b37941b9cf713ebf

Expected Result

For the services like they did with 2025-02-beta not to crash.

Actual Result

The services admin, api, identity all crash.

Adding group `bitwarden' (GID 1000) ...
Done.
Adding user `bitwarden' ...
Adding new user `bitwarden' (1000) with group `bitwarden (1000)' ...
Not creating home directory `/home/bitwarden'.
Adding new user `bitwarden' to supplemental / extra groups `users' ...
Adding user `bitwarden' to group `users' ...
...+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+..+..........+...+........+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+.....+..........+...............+......+........+......+.....................+..........+.........+...+...........+............+....+...+........+...+.+...........+.............+..+....+.....+.........+.......+..+.+...........+.........+..........+......+.....+......+...+.......+........................+.....+...+.............+............+..+....+.....+.+........+............+...+..........+......+...............+......+...+............+......+...+..+...+......+....+......+........+......+.......+..+............+.+........+.........+.......+...+.....+.............+..+...+...+.+...+.....+.+......+.....+....+.....+......+.+............+...+........+................+...........+...+.+...............+......+...+.....+...............+....+...+.....+...+.+......+...............+.....+.......+..................+...+.....+.+........+.........................+..+......+....+.........+.........+......+..+.........+...+..........+............+.....+...+...............+......+......+..........+...+...+.....+.........+...+................+.....+.........+................+............+.....+.........+......+....+..............+....+..+......+..................+.+......+..+..........+..+....+.....+.+.....+.+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
..+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.....+...+..........+...+.........+...+..+...+............+.+.....+.+...+...+...+..+.........+...+.+......+.........+.........+........+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+...+..+.+.....+......+..........+........+.......+...+...........+..........+..+.......+..+....+.................+...+.+.....+................+...+.....................+...........+..........+.....+............+.......+...+......+.....+.+...+...........+..........+.........+...........+......+.+.....+.............+..+.............+.....+.+......+.............................+..................+......+......+....+.........+...+..+.......+.....................+..+............+....+.....+...............+.......+........+....+..+.+..............+.+.................................+.........+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
2025-03-21 22:04:46,948 INFO Included extra file "/etc/supervisor.d/admin.ini" during parsing
2025-03-21 22:04:46,948 INFO Included extra file "/etc/supervisor.d/api.ini" during parsing
2025-03-21 22:04:46,948 INFO Included extra file "/etc/supervisor.d/events.ini" during parsing
2025-03-21 22:04:46,948 INFO Included extra file "/etc/supervisor.d/icons.ini" during parsing
2025-03-21 22:04:46,948 INFO Included extra file "/etc/supervisor.d/identity.ini" during parsing
2025-03-21 22:04:46,948 INFO Included extra file "/etc/supervisor.d/nginx.ini" during parsing
2025-03-21 22:04:46,948 INFO Included extra file "/etc/supervisor.d/notifications.ini" during parsing
2025-03-21 22:04:46,948 INFO Included extra file "/etc/supervisor.d/scim.ini" during parsing
2025-03-21 22:04:46,948 INFO Included extra file "/etc/supervisor.d/sso.ini" during parsing
2025-03-21 22:04:46,953 INFO RPC interface 'supervisor' initialized
2025-03-21 22:04:46,953 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2025-03-21 22:04:46,954 INFO supervisord started with pid 1
2025-03-21 22:04:47,959 INFO spawned: 'identity' with pid 65
2025-03-21 22:04:47,962 INFO spawned: 'admin' with pid 66
2025-03-21 22:04:47,968 INFO spawned: 'api' with pid 67
2025-03-21 22:04:47,971 INFO spawned: 'icons' with pid 68
2025-03-21 22:04:47,974 INFO spawned: 'nginx' with pid 69
2025-03-21 22:04:47,978 INFO spawned: 'notifications' with pid 70
2025-03-21 22:04:50,083 WARN exited: admin (terminated by SIGABRT (core dumped); not expected)
2025-03-21 22:04:50,084 WARN exited: api (terminated by SIGABRT (core dumped); not expected)
2025-03-21 22:04:51,141 INFO spawned: 'admin' with pid 145
2025-03-21 22:04:51,146 INFO spawned: 'api' with pid 146
2025-03-21 22:04:51,148 WARN exited: identity (terminated by SIGABRT (core dumped); not expected)
2025-03-21 22:04:52,160 INFO spawned: 'identity' with pid 169
2025-03-21 22:04:52,161 WARN exited: api (terminated by SIGABRT (core dumped); not expected)
2025-03-21 22:04:52,164 WARN exited: admin (terminated by SIGABRT (core dumped); not expected)
2025-03-21 22:04:54,104 WARN exited: identity (terminated by SIGABRT (core dumped); not expected)
2025-03-21 22:04:55,109 INFO spawned: 'admin' with pid 181
2025-03-21 22:04:55,112 INFO spawned: 'api' with pid 182
2025-03-21 22:04:56,097 WARN exited: api (terminated by SIGABRT (core dumped); not expected)
2025-03-21 22:04:57,107 INFO spawned: 'identity' with pid 205
2025-03-21 22:04:57,109 WARN exited: admin (terminated by SIGABRT (core dumped); not expected)
2025-03-21 22:04:59,105 INFO spawned: 'api' with pid 218
2025-03-21 22:04:59,107 WARN exited: identity (terminated by SIGABRT (core dumped); not expected)
2025-03-21 22:04:59,997 WARN exited: api (terminated by SIGABRT (core dumped); not expected)
2025-03-21 22:05:01,002 INFO spawned: 'admin' with pid 230
2025-03-21 22:05:01,004 INFO gave up: api entered FATAL state, too many start retries too quickly
2025-03-21 22:05:01,841 WARN exited: admin (terminated by SIGABRT (core dumped); not expected)
2025-03-21 22:05:02,846 INFO spawned: 'identity' with pid 242
2025-03-21 22:05:02,847 INFO gave up: admin entered FATAL state, too many start retries too quickly
2025-03-21 22:05:03,300 INFO success: icons entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
2025-03-21 22:05:03,300 INFO success: nginx entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
2025-03-21 22:05:03,300 INFO success: notifications entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
2025-03-21 22:05:03,835 WARN exited: identity (terminated by SIGABRT (core dumped); not expected)
2025-03-21 22:05:04,838 INFO gave up: identity entered FATAL state, too many start retries too quickly

Screenshots or Videos

No response

Additional Context

The the last working image (2025.2.0-beta) this works fine running the same command, and is the last version that passed my CI tests for my homelab.

$ podman run -it --rm \
  -e BW_INSTALLATION_ID=00000000-0000-0000-0000-000000000000 \
  -e BW_INSTALLATION_KEY=xxxxxxxxxxxx \
  -e BW_DOMAIN=test.local \
  -e BW_DB_PROVIDER=sqlite \
  -e globalSettings__disableUserRegistration=true \
  -p 8080 \
  docker.io/bitwarden/self-host:2025.2.0-beta@sha256:75651a95f13b9c389cb87c1e3d9a325d4344778adef88795a8cc51a7cbe85c15

Adding group `bitwarden' (GID 1000) ...
Done.
Adding user `bitwarden' ...
Adding new user `bitwarden' (1000) with group `bitwarden (1000)' ...
Not creating home directory `/home/bitwarden'.
Adding new user `bitwarden' to supplemental / extra groups `users' ...
Adding user `bitwarden' to group `users' ...
....+.+..+...+......................+..+............................+...+...+..+....+......+...............+...+.....+.+...+..+..........+.....+.........+....+...+..+...+.......+......+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...........+..+.+............+..+.+...+.................+....+.....................+..+.+.........+...........+.+......+..+.+.........+..+.+........+.+...+......+...........+...+.+..+....+...+...........+....+......+.....+....+..+.........+.......+.........+.....+..........+......+...+..+...............+....+.....+...+..........+.....+......+.......+..+.......+......+...+..+...+.+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+............+...+......+.....+...............+....+..................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
........+..+....+.....+....+......+...+...+............+..+...............+.+..+.+..+...+.......+.....+.......+......+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+.+..+....+.....+.........+.........+.+..+.......+..+.+..+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..............+.........+.....................+.........+......+................+......+..+..........+..................+.....+.......+...+..............+...+...+.............+.....+...............+.+........+..........+...............+........+.........+....+......+.....+...+..........+...............+...+..+.+...............+......+.....+.+..............................+..+...+....+.....+....+............+.....+...+................+......+.........+.....+.........+...................+..+...............+..........+........+...+...+....+............+........+...+....+......+..+..........+......+..+......................+..+....+..............+...+...+....+...+...+......+...........+...+....+..+......+.......+...+...+........................+........+..........+.................+.+.....+......+....+..............+..........+.......................+.......+............+......+...+..+.........+...............+.........+......+.+........+...................+..+.+..+.+.........+...........+.+...............+.....+.+...............+...+.....+.......+..+...+.......+........+.......+..+...................+...............+...........+.+..+.+...........+.........................+...+..+.+............+............+.....................+..+...+...................+........+..........+..+.+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
2025-03-21 21:58:04,665 INFO Included extra file "/etc/supervisor.d/admin.ini" during parsing
2025-03-21 21:58:04,665 INFO Included extra file "/etc/supervisor.d/api.ini" during parsing
2025-03-21 21:58:04,665 INFO Included extra file "/etc/supervisor.d/events.ini" during parsing
2025-03-21 21:58:04,665 INFO Included extra file "/etc/supervisor.d/icons.ini" during parsing
2025-03-21 21:58:04,665 INFO Included extra file "/etc/supervisor.d/identity.ini" during parsing
2025-03-21 21:58:04,665 INFO Included extra file "/etc/supervisor.d/nginx.ini" during parsing
2025-03-21 21:58:04,665 INFO Included extra file "/etc/supervisor.d/notifications.ini" during parsing
2025-03-21 21:58:04,665 INFO Included extra file "/etc/supervisor.d/scim.ini" during parsing
2025-03-21 21:58:04,665 INFO Included extra file "/etc/supervisor.d/sso.ini" during parsing
2025-03-21 21:58:04,669 INFO RPC interface 'supervisor' initialized
2025-03-21 21:58:04,669 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2025-03-21 21:58:04,669 INFO supervisord started with pid 1
2025-03-21 21:58:05,674 INFO spawned: 'identity' with pid 65
2025-03-21 21:58:05,676 INFO spawned: 'admin' with pid 66
2025-03-21 21:58:05,679 INFO spawned: 'api' with pid 67
2025-03-21 21:58:05,681 INFO spawned: 'icons' with pid 68
2025-03-21 21:58:05,683 INFO spawned: 'nginx' with pid 69
2025-03-21 21:58:05,686 INFO spawned: 'notifications' with pid 70
2025-03-21 21:58:21,491 INFO success: identity entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
2025-03-21 21:58:21,492 INFO success: admin entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
2025-03-21 21:58:21,492 INFO success: api entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
2025-03-21 21:58:21,493 INFO success: icons entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
2025-03-21 21:58:21,493 INFO success: nginx entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
2025-03-21 21:58:21,494 INFO success: notifications entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)

As you can see the processes all started correctly

Build Version

2025.3.0-beta

Environment

Self-Hosted

Environment Details

Docker or podman both have the same issue

Issue Tracking Info

  • [x] I understand that work is tracked outside of Github. A PR will be linked to this issue should one be opened to address it, but Bitwarden doesn't use fields like "assigned", "milestone", or "project" to track progress.

ahayzen avatar Mar 21 '25 22:03 ahayzen

Hi there,

Thanks for the report, I have escalated this for further investigation. If you have more information that can help us, please add it below.

Thanks!

Greenderella avatar Mar 25 '25 11:03 Greenderella

@Greenderella I have similar issue (SIGKILL instead of SIGABRT). Works for about 2 minutes after starting and starts throwing warnings and becomes unresponsive.

2025-04-01 14:16:18,287 INFO RPC interface 'supervisor' initialized
2025-04-01 14:16:18,287 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2025-04-01 14:16:18,288 INFO supervisord started with pid 1
2025-04-01 14:16:19,292 INFO spawned: 'identity' with pid 66
2025-04-01 14:16:19,298 INFO spawned: 'admin' with pid 67
2025-04-01 14:16:19,301 INFO spawned: 'api' with pid 68
2025-04-01 14:16:19,304 INFO spawned: 'icons' with pid 69
2025-04-01 14:16:19,316 INFO spawned: 'nginx' with pid 70
2025-04-01 14:16:19,330 INFO spawned: 'notifications' with pid 71
2025-04-01 14:16:34,445 INFO success: identity entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
2025-04-01 14:16:34,447 INFO success: admin entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
2025-04-01 14:16:34,447 INFO success: api entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
2025-04-01 14:16:34,447 INFO success: icons entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
2025-04-01 14:16:34,447 INFO success: nginx entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
2025-04-01 14:16:34,447 INFO success: notifications entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
2025-04-01 14:18:03,846 WARN exited: api (terminated by SIGKILL; not expected)
2025-04-01 14:18:04,008 INFO spawned: 'api' with pid 207
2025-04-01 14:18:19,930 INFO success: api entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
2025-04-01 14:24:07,869 WARN exited: identity (terminated by SIGKILL; not expected)
2025-04-01 14:24:08,898 INFO spawned: 'identity' with pid 380
2025-04-01 14:24:24,167 INFO success: identity entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
2025-04-01 14:55:01,417 WARN exited: identity (terminated by SIGKILL; not expected)
2025-04-01 14:55:02,438 INFO spawned: 'identity' with pid 704
2025-04-01 14:55:18,134 INFO success: identity entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
2025-04-01 15:00:08,208 WARN exited: api (terminated by SIGKILL; not expected)
2025-04-01 15:00:09,225 INFO spawned: 'api' with pid 755
2025-04-01 15:00:24,997 INFO success: api entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
2025-04-01 15:48:37,174 WARN exited: identity (terminated by SIGKILL; not expected)
2025-04-01 15:48:38,187 INFO spawned: 'identity' with pid 978``` 



Pulled the instance last week

``` beta: Pulling from bitwarden/self-host
Digest: sha256:f4e9e8122fe08d7e8e5f30edc7ed0091de166151b797fae7fe45b190766ef125
Status: Image is up to date for bitwarden/self-host:beta
docker.io/bitwarden/self-host:beta```

Bramzor avatar Apr 01 '25 15:04 Bramzor

OK, I have figured this out by execing into the running container and looking at the logs.

The following happens with the newer versions of bitwarden

Unhandled exception. System.InvalidOperationException: Installation Id must be set for self-hosted installations.
   at Bit.SharedWeb.Utilities.ServiceCollectionExtensions.AddDefaultServices(IServiceCollection services, GlobalSettings globalSettings) in /source/src/SharedWeb/Utilities/ServiceCollectionExtensions.cs:line 287
   at Bit.Identity.Startup.ConfigureServices(IServiceCollection services) in /source/src/Identity/Startup.cs:line 147
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
   at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(Type startupType, HostBuilderContext context, IServiceCollection services, Object instance)
   at Microsoft.Extensions.Hosting.HostBuilder.InitializeServiceProvider()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at Bit.Identity.Program.Main(String[] args) in /source/src/Identity/Program.cs:line 10

You can see the error being thrown here

https://github.com/bitwarden/server/blob/0bad7a6e5f265b5bcfeacd77b489e9f0cc21739c/src/SharedWeb/Utilities/ServiceCollectionExtensions.cs#L283-L288

So it appears that you must now have a valid installation id to start the services, this was changed as part of https://github.com/bitwarden/server/commit/9f4aa1ab2bcda4aaef9dcf25350a1f3f78eb3417 which matches when this broke for me.

Changing to a valid installation id then allows this to continue, therefore i am closing this issue.

ahayzen avatar Apr 05 '25 21:04 ahayzen