sqld icon indicating copy to clipboard operation
sqld copied to clipboard
verified publisher icon libsql

Allow sqld graceful shutdown

Open george-lim opened this issue 2 years ago • 4 comments

When using sqld with docker compose as specified, pressing Ctrl+C while the containers are running will not result in sqld gracefully shutdown. Instead, it will hang for 10 seconds until docker sends a SIGKILL.

When using bottomless, all successive runs will always start with:

INFO sqld::replication::primary::logger: Replication log is dirty, recovering from database file.

Repro steps:

  1. Choose either docker compose file from here and start it
  2. After everything is initialized, press Ctrl+C to gracefully shut down the containers
  3. Observe that writer and reader each timeout after 10 seconds, taking 20 seconds total while nginx shuts down immediately

george-lim avatar Jul 29 '23 21:07 george-lim

@george-lim I'm not sure what version you are using, but isn't that line being logged when you send the signal:

2023-08-28T13:35:42.222280Z  INFO sqld: received CTRL-C, shutting down gracefully... This may take some time

MarinPostma avatar Aug 28 '23 13:08 MarinPostma

No I don't get that line when I press Ctrl+C even on the latest version 0.20.2. Perhaps it's due to my configuration here?

services:
  minio:
    image: minio/minio:RELEASE.2023-08-04T17-40-21Z.hotfix.04968f7ec
    command: server /data
    volumes:
      - minio_data:/data
    expose:
      - 9000
  sqld:
    image: ghcr.io/libsql/sqld:v0.20.2
    environment:
      - SQLD_NODE=standalone
      - SQLD_ENABLE_BOTTOMLESS_REPLICATION=true
      - LIBSQL_BOTTOMLESS_ENDPOINT=http://minio:9000
      - AWS_ACCESS_KEY_ID=minioadmin
      - AWS_SECRET_ACCESS_KEY=minioadmin
      - AWS_DEFAULT_REGION=us-east-1
    volumes:
      - sqld_data:/var/lib/sqld
    expose:
      - 5001
      - 8080
    depends_on:
      - minio
Screenshot 2023-09-06 at 6 52 37 PM

george-lim avatar Sep 06 '23 22:09 george-lim

I'm going to compile the docker compose image again from main and test it by pressing Ctrl+C, perhaps it was fixed since I last investigated this.

george-lim avatar Sep 06 '23 22:09 george-lim

@MarinPostma Nope, confirmed does not work. How did you get that message to show up? This is what I see when I pull main, and execute docker compose build && docker compose up from the docker-compose folder.

george@Stealth docker-compose % docker compose up                                                                                                                                  [main|✔]
[+] Running 3/0
 ✔ Container docker-compose-writer-1  Created                                                                                                                                          0.0s 
 ✔ Container docker-compose-reader-1  Created                                                                                                                                          0.0s 
 ✔ Container docker-compose-nginx-1   Created                                                                                                                                          0.0s 
Attaching to docker-compose-nginx-1, docker-compose-reader-1, docker-compose-writer-1
docker-compose-writer-1  |            _     _ 
docker-compose-writer-1  |  ___  __ _| | __| |
docker-compose-writer-1  | / __|/ _` | |/ _` |
docker-compose-writer-1  | \__ \ (_| | | (_| |
docker-compose-writer-1  | |___/\__, |_|\__,_|
docker-compose-writer-1  |         |_|        
docker-compose-writer-1  | 
docker-compose-writer-1  | Welcome to sqld!
docker-compose-writer-1  | 
docker-compose-writer-1  | 
docker-compose-writer-1  | version: 0.20.2
docker-compose-writer-1  | commit SHA: ef55476970d1bec1f157515c3e41eb9817725d15
docker-compose-writer-1  | build date: 2023-09-06
docker-compose-writer-1  | 
docker-compose-writer-1  | This software is in BETA version.
docker-compose-writer-1  | If you encounter any bug, please open an issue at https://github.com/libsql/sqld/issues
docker-compose-writer-1  | 
docker-compose-writer-1  | config:
docker-compose-writer-1  |      - mode: primary (0.0.0.0:5001)
docker-compose-writer-1  |      - database path: iku.db
docker-compose-writer-1  |      - extensions path: <disabled>
docker-compose-writer-1  |      - listening for HTTP requests on: 0.0.0.0:8080
docker-compose-writer-1  |      - grpc_tls: no
docker-compose-writer-1  | 
docker-compose-writer-1  | 2023-09-06T22:58:09.928320Z  WARN sqld: No authentication specified, the server will not require authentication
docker-compose-writer-1  | 2023-09-06T22:58:09.928426Z  WARN sqld: No authentication specified, the server will not require authentication
docker-compose-writer-1  | 2023-09-06T22:58:09.928547Z  INFO sqld::rpc: serving write proxy server at 0.0.0.0:5001
docker-compose-writer-1  | 2023-09-06T22:58:09.929112Z  WARN sqld: No server heartbeat configured
docker-compose-writer-1  | 2023-09-06T22:58:09.929101Z  INFO sqld::http: listening for HTTP requests on 0.0.0.0:8080
docker-compose-reader-1  |            _     _ 
docker-compose-reader-1  |  ___  __ _| | __| |
docker-compose-reader-1  | / __|/ _` | |/ _` |
docker-compose-reader-1  | \__ \ (_| | | (_| |
docker-compose-reader-1  | |___/\__, |_|\__,_|
docker-compose-reader-1  |         |_|        
docker-compose-reader-1  | 
docker-compose-reader-1  | Welcome to sqld!
docker-compose-reader-1  | 
docker-compose-reader-1  | version: 0.20.2
docker-compose-reader-1  | commit SHA: ef55476970d1bec1f157515c3e41eb9817725d15
docker-compose-reader-1  | build date: 2023-09-06
docker-compose-reader-1  | 
docker-compose-reader-1  | This software is in BETA version.
docker-compose-reader-1  | If you encounter any bug, please open an issue at https://github.com/libsql/sqld/issues
docker-compose-reader-1  | 
docker-compose-reader-1  | config:
docker-compose-reader-1  |      - mode: replica (primary at http://writer:5001)
docker-compose-reader-1  |      - database path: iku.db
docker-compose-reader-1  |      - extensions path: <disabled>
docker-compose-reader-1  |      - listening for HTTP requests on: 0.0.0.0:8080
docker-compose-reader-1  |      - grpc_tls: no
docker-compose-reader-1  | 
docker-compose-reader-1  | 2023-09-06T22:58:10.115799Z  WARN sqld: No authentication specified, the server will not require authentication
docker-compose-reader-1  | 2023-09-06T22:58:10.116052Z  WARN sqld: No server heartbeat configured
docker-compose-reader-1  | 2023-09-06T22:58:10.116084Z  INFO sqld::http: listening for HTTP requests on 0.0.0.0:8080
docker-compose-nginx-1   | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
docker-compose-nginx-1   | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
docker-compose-nginx-1   | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
docker-compose-nginx-1   | 10-listen-on-ipv6-by-default.sh: info: IPv6 listen already enabled
docker-compose-nginx-1   | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
docker-compose-nginx-1   | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
docker-compose-nginx-1   | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
docker-compose-nginx-1   | /docker-entrypoint.sh: Configuration complete; ready for start up
^CGracefully stopping... (press Ctrl+C again to force)
Aborting on container exit...
[+] Stopping 3/3
 ✔ Container docker-compose-nginx-1   Stopped                                                                                                                                               0.2s 
 ✔ Container docker-compose-reader-1  Stopped                                                                                                                                              10.1s 
 ✔ Container docker-compose-writer-1  Stopped                                                                                                                                              10.1s 
canceled
george@Stealth docker-compose %

Notice how bot hthe reader and the writer took 10s to stop. This is because Docker sends a SIGKILL after 10s timeout

george-lim avatar Sep 06 '23 23:09 george-lim