Sentry shutdown by using backup.sh script

[codegain]

Self-Hosted Version

23.11.2

CPU Architecture

x86_64

Docker Version

23.0.1

Docker Compose Version

2.16.0

Steps to Reproduce

1. Run a normal self-hosted sentry
2. Execute ./scripts/backup.sh

Expected Result

Backup created and sentry keeps running like with the "old" backup method:

docker compose run -v $(pwd)/sentry:/sentry-data/backup  --rm -T -e SENTRY_LOG_LEVEL=CRITICAL web export /sentry-data/backup/backup.json

Actual Result

Backup created but sentry shutting down and not starting again after script finished. Must be manually started.

Event ID

No response

[codegain]

If you only copy the backup command from the _lib.sh

https://github.com/getsentry/self-hosted/blob/bf66ddd2241cb5d2e986abcab1e307b5563efe21/scripts/_lib.sh#L65

Sentry keeps running - only if you execute the backup.sh script it shuts itself down. Is this intentional?

[azaslavsky]

This is very odd. Can you set -x before running the backup.sh version that does not work and the _lib.sh version that does, and report back if you detect any differences?

Also, is the underlying backup itself working?

[codegain]

@azaslavsky I get back to you after I work on our sentry setup again - but I can't promise a date before the new year.

[sblaisot]

I can confirm this behaviour with 23.11.2 self hosted.

There is definitely something wrong when sourcing _lib.sh

root@sentry:/opt/sentry# cmd=backup
root@sentry:/opt/sentry# source scripts/_lib.sh 
▶ Initializing Docker Compose ...
basename: invalid option -- 'b'
Try 'basename --help' for more information.

▶ Detecting Docker platform
Detected Docker platform is linux/amd64

▶ Setting up error handling ...
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            Install the buildx component to build images with BuildKit:
            https://docs.docker.com/go/buildx/

Sending build context to Docker daemon  2.048kB
Step 1/4 : FROM debian:bookworm-slim
 ---> 31d5e503c34f
Step 2/4 : LABEL MAINTAINER="[email protected]"
 ---> Using cache
 ---> 221b31e88e07
Step 3/4 : RUN set -x   && apt-get update   && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends jq   && apt-get clean   && rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> c2081cd8bd2f
Step 4/4 : ENTRYPOINT ["jq"]
 ---> Using cache
 ---> 6357b792d868
Successfully built 6357b792d868
Successfully tagged sentry-self-hosted-jq-local:latest

Hey, so ... we would love to automatically find out about issues with your
Sentry instance so that we can improve the product. Turns out there is an app
for that, called Sentry. Would you be willing to let us automatically send data
about your instance upstream to Sentry for development and debugging purposes?

  y / yes / 1
  n / no / 0

(Btw, we send this to our own self-hosted Sentry instance, not to Sentry SaaS,
so that we can be in this together.)

Here's the info we may collect:

  - OS username
  - IP address
  - install log
  - runtime errors
  - performance data

Thirty (30) day retention. No marketing. Privacy policy at sentry.io/privacy.

y or n? ^C

Notice :

• basename: invalid option -- 'b'
• .env file not taken in account (I have REPORT_SELF_HOSTED_ISSUES=0 in my .env file)

I can see at the end of _lib.sh that other scripts from install directory are sourced.

[sblaisot]

ok, basename: invalid option -- 'b' is probably because I sourced it from an interactive shell so here $0 is -bash

[sblaisot]

from https://github.com/getsentry/self-hosted/blob/master/scripts/_lib.sh#L94 I can see that on EXIT signal (i.e. end of script, whatever the reason), the cleanup() function from install/error-handling.sh is getting called and this function calls docker compose stop here if the variable MINIMIZE_DOWNTIME is empty or unset

[sblaisot]

As a workaround, one can make a backup without sentry going down with

MINIMIZE_DOWNTIME=1 REPORT_SELF_HOSTED_ISSUES=0 scripts/backup.sh

Also sentry/backup.json should probably be added to .gitignore

[hubertdeng123]

Thanks for chiming in with your solution @sblaisot. You shouldn't need to set the MINIMIZE_DOWNTIME but glad you were able to find a workaround. Could you give some more information about your setup, such as your platform you're running self-hosted on?

[sblaisot]

I'm running sentry 23.11.2 self hosted on Debian 12 with Docker 24.0.7

root@sentry:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 12 (bookworm)
Release:	12
Codename:	bookworm

root@sentry:~# docker version
Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:08:02 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:02 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.26
  GitCommit:        3dd1e886e55dd695541fdcd67420c2888645a495
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

root@sentry:~# docker compose version
Docker Compose version v2.21.0

[hubertdeng123]

Thanks for the info. We'll keep this on our backlog for now