upgrading from 10.1 to 21.5.0

[jreagan000]

Self-Hosted Version

Sentry 10.1.0.dev0

CPU Architecture

x86_64

Docker Version

Docker version 27.3.1, build ce12230

Docker Compose Version

docker-compose version 1.29.2, build 5becea4c

Steps to Reproduce

run sudo ./install.sh

Expected Result

application successfully installed

Actual Result

I am having issues upgrading from 10.1 to 21.5.0. I am using docker on Ubuntu 20.04 I am getting the following and not sure how to get passed it, its trying to pull from a dead URL since stretch has moved to the archives but i cant figure out how to have this reference the archives. I updated /etc/apt/sources.list but it doesnt seem to do anything to this install script.

https://github.com/getsentry/self-hosted/pull/6 1.049 Ign:12 http://deb.debian.org/debian stretch-updates/main all Packages https://github.com/getsentry/self-hosted/pull/6 1.060 Ign:9 http://deb.debian.org/debian stretch/main all Packages https://github.com/getsentry/self-hosted/pull/6 1.083 Err:10 http://deb.debian.org/debian stretch/main amd64 Packages https://github.com/getsentry/self-hosted/pull/6 1.083 404 Not Found https://github.com/getsentry/self-hosted/pull/6 1.188 Err:11 http://deb.debian.org/debian stretch-updates/main amd64 Packages https://github.com/getsentry/self-hosted/pull/6 1.188 404 Not Found https://github.com/getsentry/self-hosted/pull/6 1.200 Ign:12 http://deb.debian.org/debian stretch-updates/main all Packages https://github.com/getsentry/self-hosted/pull/6 1.206 Reading package lists... https://github.com/getsentry/self-hosted/pull/6 1.218 W: The repository 'http://security.debian.org/debian-security stretch/updates Release' does not have a Release file. https://github.com/getsentry/self-hosted/pull/6 1.218 W: The repository 'http://deb.debian.org/debian stretch Release' does not have a Release file. https://github.com/getsentry/self-hosted/pull/6 1.218 W: The repository 'http://deb.debian.org/debian stretch-updates Release' does not have a Release file. https://github.com/getsentry/self-hosted/pull/6 1.218 E: Failed to fetch http://security.debian.org/debian-security/dists/stretch/updates/main/binary-amd64/Packages 404 Not Found [IP: 151.101.2.132 80] https://github.com/getsentry/self-hosted/pull/6 1.218 E: Failed to fetch http://deb.debian.org/debian/dists/stretch/main/binary-amd64/Packages 404 Not Found https://github.com/getsentry/self-hosted/pull/6 1.218 E: Failed to fetch http://deb.debian.org/debian/dists/stretch-updates/main/binary-amd64/Packages 404 Not Found https://github.com/getsentry/self-hosted/pull/6 1.218 E: Some index files failed to download. They have been ignored, or old ones used instead. https://github.com/getsentry/self-hosted/pull/6 ERROR: process "/bin/sh -c apt-get update && apt-get install -y --no-install-recommends cron && rm -r /var/lib/apt/lists/*" did not complete successfully: exit code: 100

Event ID

ERROR: process "/bin/sh -c apt-get update && apt-get install -y --no-install-recommends cron && rm -r /var/lib/apt/lists/*" did not complete successfully: exit code: 100

#1 WARN: InvalidDefaultArgInFrom: Default value for ARG ${BASE_IMAGE} results in empty or invalid base image name (line 2)

[aldy505]

I updated /etc/apt/sources.list but it doesnt seem to do anything to this install script.

Exactly how did you update the /etc/apt/sources.list?

But that question aside, you'd need to modify the value of /etc/apt/sources.list on a few Dockerfiles: https://github.com/getsentry/self-hosted/blob/ee240f041c976df9e4701f80a0f6f2dd9b3fdd7b/cron/Dockerfile#L1-L6

ARG BASE_IMAGE
FROM ${BASE_IMAGE}
RUN printf "deb http://deb.debian.org/debian stretch main\ndeb http://archive.debian.org/debian-security stretch/updates main\n" > /etc/apt/sources.list
USER 0
RUN if [ -n "${http_proxy}" ]; then echo "Acquire::http::proxy \"${http_proxy}\";" >> /etc/apt/apt.conf; fi
RUN if [ -n "${https_proxy}" ]; then echo "Acquire::https::proxy \"${https_proxy}\";" >> /etc/apt/apt.conf; fi

Then re-run the ./install.sh script

[jreagan000]

I updated /etc/apt/sources.list but it doesnt seem to do anything to this install script.

Exactly how did you update the /etc/apt/sources.list?

But that question aside, you'd need to modify the value of /etc/apt/sources.list on a few Dockerfiles:

self-hosted/cron/Dockerfile

Lines 1 to 6 in ee240f0

ARG BASE_IMAGE FROM ${BASE_IMAGE} RUN apt-get update && apt-get install -y --no-install-recommends cron && \ rm -r /var/lib/apt/lists/* COPY entrypoint.sh /entrypoint.sh ENTRYPOINT ["/entrypoint.sh"] ARG BASE_IMAGE FROM ${BASE_IMAGE} RUN printf "deb http://deb.debian.org/debian stretch main\ndeb http://archive.debian.org/debian-security stretch/updates main\n" > /etc/apt/sources.list USER 0 RUN if [ -n "${http_proxy}" ]; then echo "Acquire::http::proxy "${http_proxy}";" >> /etc/apt/apt.conf; fi RUN if [ -n "${https_proxy}" ]; then echo "Acquire::https::proxy "${https_proxy}";" >> /etc/apt/apt.conf; fi Then re-run the ./install.sh script

I had used this command to update it and added the following lines:

sudo nano /etc/apt/sources.list

deb [trusted=yes] https://archive.debian.org/debian/ stretch main #deb-src http://deb.debian.org/debian/ stretch main deb [trusted=yes] https://archive.debian.org/debian-security/ stretch/updates main #deb-src http://security.debian.org/ stretch/updates main deb [trusted=yes] https://archive.debian.org/debian/ stretch-proposed-updates main #deb-src http://deb.debian.org/debian/ stretch-updates main

But your updated Dockerfile worked perfectly, Thank you!

---

Question #2 I have one more question for this upgrade, hopefully. Some additional details. This is all part of a migration from GCP to AWS and in the process i am updating the OS to a newer version of ubuntu since it is running on a very old debian release. On the old instance we have the data and files on a secondary disk that is mounted to /var/storage/. After the install i need to run "docker-compose up -d" which i assume references the docker-compose.yml file in the same folder as the install.sh.

The previous docker-compose.yml file had these lines in it but from the image name this looks to be a very old yml.

services:

base: restart: unless-stopped image: sentry:8.21-onbuild environment: # Run docker-compose run web config generate-secret-key # to get the SENTRY_SECRET_KEY value. SENTRY_SECRET_KEY: "xxxxxxxxxxxxxxxxxxxx" SENTRY_MEMCACHED_HOST: memcached SENTRY_REDIS_HOST: redis SENTRY_POSTGRES_HOST: postgres SENTRY_EMAIL_HOST: "xxxxxxxxxxxxxx" SENTRY_EMAIL_PORT: xxxxxxxxxx SENTRY_EMAIL_USER: "xxxxxxxxxxxxxx" SENTRY_EMAIL_PASSWORD: "xxxxxxxxxxxxxxx" SENTRY_EMAIL_USE_TLS: 1 SENTRY_SERVER_EMAIL: "xxxxxxxxxxxxxxxx" SENTRY_USE_SSL: 1 volumes: - /var/storage/services/sentry/files/:/var/lib/sentry/files

memcached: restart: unless-stopped image: memcached:1.4

redis: restart: unless-stopped image: redis:3.2-alpine entrypoint: redis-server --save ""

postgres: restart: unless-stopped image: postgres:9.5 volumes: - /var/storage/services/sentry/data/:/var/lib/postgresql/data -

So on the new ubuntu instance i have made a snapshot of the secondary disk and mounted a copy of it to this new instance. In the yaml file what do i need to do to make the new instance the same config or use the secondary disks file and data location. The data one seems straightforward enough since that looks to be the postgres data but the config under the senrtry image is whats throwing me off since there doesnt seem to be a spot in the yaml thats similar.

[aldy505]

SENTRY_EMAIL_HOST: "xxxxxxxxxxxxxx"
SENTRY_EMAIL_PORT: xxxxxxxxxx
SENTRY_EMAIL_USER: "xxxxxxxxxxxxxx"
SENTRY_EMAIL_PASSWORD: "xxxxxxxxxxxxxxx"
SENTRY_EMAIL_USE_TLS: 1

These are moved here https://github.com/getsentry/self-hosted/blob/5910c02cc404e533348de84d99acb6d9f7b85530/sentry/config.example.yml#L11-L16

SENTRY_SERVER_EMAIL: "xxxxxxxxxxxxxxxx"

Also moved here: https://github.com/getsentry/self-hosted/blob/5910c02cc404e533348de84d99acb6d9f7b85530/sentry/config.example.yml#L23-L25

SENTRY_SECRET_KEY: "xxxxxxxxxxxxxxxxxxxx"

Moved here: https://github.com/getsentry/self-hosted/blob/5910c02cc404e533348de84d99acb6d9f7b85530/sentry/config.example.yml#L50

SENTRY_MEMCACHED_HOST: memcached
SENTRY_REDIS_HOST: redis
SENTRY_POSTGRES_HOST: postgres

No need to think about these, but if you want to, they're hardcoded as default values here:

• https://github.com/getsentry/self-hosted/blob/5910c02cc404e533348de84d99acb6d9f7b85530/sentry/sentry.conf.example.py#L51
• https://github.com/getsentry/self-hosted/blob/5910c02cc404e533348de84d99acb6d9f7b85530/sentry/sentry.conf.example.py#L84
• https://github.com/getsentry/self-hosted/blob/5910c02cc404e533348de84d99acb6d9f7b85530/sentry/sentry.conf.example.py#L117

[jreagan000]

So all of the docker containers are running now but these three are erroring out.

The symbolicator is the only one with errors being shown:

error: Found argument '55 23 * * * gosu symbolicator symbolicator cleanup' which wasn't expected, or isn't valid in this context

USAGE: symbolicator [OPTIONS] <SUBCOMMAND>

For more information try --help

The two snuba containers just show this:

Usage: snuba [OPTIONS] COMMAND [ARGS]...

                   o
                  O
                  O
                  o

.oOo 'OoOo. O o OoOo. .oOoO' Ooo. o O o O O o O o O O o O o o O o O OoO' o O OoO'o OoO' `OoO'o

Options: --version Show the version and exit. --help Show this message and exit.

Commands: api bootstrap Warning: Not intended to be used in production yet. bulk-load cleanup Deletes stale partitions for ClickHouse tables config Manage runtime configuration. confirm-load Confirms the snapshot has been loaded by sending the... consumer devserver Starts all Snuba processes for local development. migrations optimize replacer subscriptions Evaluates subscribed queries for a dataset.

[jreagan000]

Saw this too during the docker compose build, looks like maybe postgres isnt starting\building

[jreagan000]

NVM the postgres errors. I fixed that by re-running install.sh after making the config changes and rerunning sudo docker-compose up -d

Those 3 containers are still failing the same way but the application is up at least. But i am not able to log into it with my credentials so im concerned the DB may not be connected properly

[jreagan000]

I was able to create a new user from the CLI and login with that user and none of my settings, users, or data is there

[jreagan000]

I fixed the postgres issue, Just had to map the volume correctly in the docker compose yaml

correct syntax: (local_host_storage_path):/var/lib/postgresql/data

So everything looks to be functional now, Not sure what those three containers that are failing do or if they are important but they are failing still

[aldy505]

Can you copy and paste the container config on the docker-compose.yml for that *-cleanup` containers?

[jreagan000]

symbolicator-cleanup: <<: *restart_policy image: symbolicator-cleanup-onpremise-local build: context: ./cron args: BASE_IMAGE: "$SYMBOLICATOR_IMAGE" command: '"55 23 * * * gosu symbolicator symbolicator cleanup"' volumes: - "sentry-symbolicator:/data"

snuba-cleanup: <<: snuba_defaults image: snuba-cleanup-onpremise-local build: context: ./cron args: BASE_IMAGE: "$SNUBA_IMAGE" command: '"/5 * * * * gosu snuba snuba cleanup --storage errors --dry-run False"'

snuba-transactions-cleanup: <<: snuba_defaults image: snuba-cleanup-onpremise-local build: context: ./cron args: BASE_IMAGE: "$SNUBA_IMAGE" command: '"/5 * * * * gosu snuba snuba cleanup --storage transactions --dry-run False"'

[aldy505]

Ah, those containers does not exists on newer versions. If you're still want to upgrade to the next version after hitting 21.5.0, you can delete those containers, you should be good to go while not having those containers running.

The only cleanup containers on newer versions are:

• https://github.com/getsentry/self-hosted/blob/8887a82a86153d0c9696463c0525976cf8e3f136/docker-compose.yml#L318-L327
• https://github.com/getsentry/self-hosted/blob/8887a82a86153d0c9696463c0525976cf8e3f136/docker-compose.yml#L443-L451
• https://github.com/getsentry/self-hosted/blob/8887a82a86153d0c9696463c0525976cf8e3f136/docker-compose.yml#L500-L516

[jreagan000]

A couple of big issues now:

Trying to upgrade to 21.6.3 i got this error:

could not translate host name "postgres" to address: Temporary failure in name resolution

---

After reverting back to 21.5.0 by running sudo docker compose up -d postgres container wont start and has the following error:

PostgreSQL Database directory appears to contain a database; Skipping initialization

LOG: invalid authentication method "trusthost" CONTEXT: line 94 of configuration file "/var/lib/postgresql/data/pg_hba.conf" FATAL: could not load pg_hba.conf LOG: database system is shut down

I have to restore the Postgres Files from Backup and rerun the 21.5.0 install to get things running again

[jreagan000]

Also it seems with the move from GCP to AWS it looks like ive lost all the event history. All of the config, project, users, etc is all there but no history of errors. Is that data somewhere besides Postgres?

[aldy505]

Also it seems with the move from GCP to AWS it looks like ive lost all the event history. All of the config, project, users, etc is all there but no history of errors. Is that data somewhere besides Postgres?

Yes the events are both on Postgres (for raw events) and Clickhouse (for processed events that you can see on the UI)

[getsantry[bot]]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀