linux packaged pomerium instance takes a long time to restart

[gaurdro]

What happened?

systemctl restart pomerium takes a very long time.

What did you expect to happen?

a restart happens very quickly

How'd it happen?

install debian packaged pomerium core systemctl start pomerium systemctl restart pomerium

What's your environment like?

• Pomerium version (retrieve with pomerium --version):

pomerium: 0.26.1-1719873919+eb8dc899 envoy: 1.30.3+6cee65d679a80a0afb018658aa10aa21c939965dca341db80935931c5b6e5c7b

• Server Operating System/Architecture/Cloud: PRETTY_NAME="Ubuntu 24.04 LTS" NAME="Ubuntu" VERSION_ID="24.04" VERSION="24.04 LTS (Noble Numbat)" VERSION_CODENAME=noble ID=ubuntu ID_LIKE=debian

What's your config.yaml?

Default from the package, but should be a zero connected instance.

# Required settings below.  See complete documentation at https://www.pomerium.com/reference/

# To run on :443 set AmbientCapabilities=CAP_NET_BIND_SERVICE
# in a systemd override
address: :8443

authenticate_service_url: https://authenticate.localhost.pomerium.io
certificates:
  - cert: /etc/pomerium/cert.pem
    key: /etc/pomerium/key.pem
shared_secret: XXXXXX
cookie_secret: YYYYY
idp_provider: "google"
idp_client_id: XXXX
idp_client_secret: YYYY

routes:
  - from: https://yoursite.localhost.pomerium.io
    to: https://yoursite.local
    policy:
      - allow:
          or:
            - user:
                is: [email protected]

What did you see in the logs?

Aug 22 00:08:18 mini-mi pomerium[2737189]: {"level":"info","signal":"terminated","time":"2024-08-22T00:08:18Z","message":"quitting..."}
Aug 22 00:08:18 mini-mi pomerium[2737189]: {"level":"error","service":"envoy","name":"main","time":"2024-08-22T00:08:18Z","message":"caught ENVOY_SIGTERM"}
Aug 22 00:08:18 mini-mi pomerium[2737189]: {"level":"error","service":"identity_manager","syncer_id":"identity_manager/users","syncer_type":"type.googleapis.com/user.User","error":"error receiving sync record: rpc error: code = Canceled desc = context canceled","time":"2024-08-22T00:08:18Z","message":"sync"}
Aug 22 00:08:18 mini-mi pomerium[2737189]: {"level":"error","service":"identity_manager","syncer_id":"identity_manager/sessions","syncer_type":"type.googleapis.com/session.Session","error":"error receiving sync record: rpc error: code = Canceled desc = context canceled","time":"2024-08-22T00:08:18Z","message":"sync"}
Aug 22 00:08:18 mini-mi systemd[1]: Stopping pomerium.service - Pomerium...
Aug 22 00:08:18 mini-mi pomerium[2737189]: {"level":"error","error":"load metric state: rpc error: code = NotFound desc = record not found","metric":"dau","time":"2024-08-22T00:08:18Z","message":"error loading metric state"}
Aug 22 00:08:18 mini-mi pomerium[2737189]: {"level":"warn","error":"rpc error: code = Canceled desc = context canceled","lease_name":"identity_manager","time":"2024-08-22T00:08:18Z","message":"leaser: error acquiring lease"}
Aug 22 00:08:18 mini-mi pomerium[2737189]: {"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","error":"error receiving sync record: rpc error: code = Unavailable desc = error reading from server: EOF","time":"2024-08-22T00:08:18Z","message":"sync"}
Aug 22 00:08:18 mini-mi pomerium[2737189]: {"level":"error","service":"envoy","name":"config","time":"2024-08-22T00:08:18Z","message":"DeltaAggregatedResources gRPC config stream to pomerium-control-plane-grpc closed: 13, "}
Aug 22 00:08:19 mini-mi pomerium[2737189]: {"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","error":"error calling sync: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:37281: connect: connection refused\"","time":"2024-08-22T00:08:19Z","message":"sync"}
Aug 22 00:08:19 mini-mi pomerium[2737189]: {"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","error":"error calling sync: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:37281: connect: connection refused\"","time":"2024-08-22T00:08:19Z","message":"sync"}
Aug 22 00:08:21 mini-mi pomerium[2737189]: {"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","error":"error calling sync: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:37281: connect: connection refused\"","time":"2024-08-22T00:08:21Z","message":"sync"}
Aug 22 00:08:23 mini-mi pomerium[2737189]: {"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","error":"error calling sync: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:37281: connect: connection refused\"","time":"2024-08-22T00:08:23Z","message":"sync"}
Aug 22 00:08:23 mini-mi pomerium[2737189]: {"level":"error","error":"load metric state: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:44401: connect: connection refused\"","metric":"mau","time":"2024-08-22T00:08:23Z","message":"error loading metric state"}
Aug 22 00:08:27 mini-mi pomerium[2737189]: {"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","error":"error calling sync: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:37281: connect: connection refused\"","time":"2024-08-22T00:08:27Z","message":"sync"}
Aug 22 00:08:30 mini-mi pomerium[2737189]: {"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","error":"error calling sync: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:37281: connect: connection refused\"","time":"2024-08-22T00:08:30Z","message":"sync"}
Aug 22 00:08:35 mini-mi pomerium[2737189]: {"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","error":"error calling sync: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:37281: connect: connection refused\"","time":"2024-08-22T00:08:35Z","message":"sync"}
Aug 22 00:08:44 mini-mi pomerium[2737189]: {"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","error":"error calling sync: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:37281: connect: connection refused\"","time":"2024-08-22T00:08:44Z","message":"sync"}
Aug 22 00:08:53 mini-mi pomerium[2737189]: {"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","error":"error calling sync: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:37281: connect: connection refused\"","time":"2024-08-22T00:08:53Z","message":"sync"}
Aug 22 00:09:13 mini-mi pomerium[2737189]: {"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","error":"error calling sync: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:37281: connect: connection refused\"","time":"2024-08-22T00:09:13Z","message":"sync"}
Aug 22 00:09:47 mini-mi pomerium[2737189]: {"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","error":"error calling sync: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:37281: connect: connection refused\"","time":"2024-08-22T00:09:47Z","message":"sync"}
Aug 22 00:09:48 mini-mi systemd[1]: pomerium.service: State 'stop-sigterm' timed out. Killing.
Aug 22 00:09:48 mini-mi systemd[1]: pomerium.service: Killing process 2737189 (pomerium) with signal SIGKILL.
Aug 22 00:09:48 mini-mi systemd[1]: pomerium.service: Failed to kill control group /system.slice/pomerium.service, ignoring: Invalid argument
Aug 22 00:09:48 mini-mi systemd[1]: pomerium.service: Main process exited, code=killed, status=9/KILL
Aug 22 00:09:48 mini-mi systemd[1]: pomerium.service: Failed with result 'timeout'.
Aug 22 00:09:48 mini-mi systemd[1]: Stopped pomerium.service - Pomerium.

Additional context

Add any other context about the problem here.