self-hosted icon indicating copy to clipboard operation
self-hosted copied to clipboard

Failure due to - Redis is loading the dataset in memory

Open klemen-df opened this issue 1 year ago • 5 comments

Self-Hosted Version

24.9.0

CPU Architecture

x86_64

Docker Version

27.3.1

Docker Compose Version

2.29.7

Steps to Reproduce

  1. docker compose up -d

Expected Result

Sentry should start successfully

Actual Result

sentry-self-hosted-web-1 fails with 'sentry.exceptions.InvalidConfiguration: Redis is loading the dataset in memory'

I think this is due to:

  redis:
    <<: *restart_policy
    image: "redis:6.2.14-alpine"
    healthcheck:
      <<: *healthcheck_defaults
      **test: redis-cli ping**

So, in the test, we only test if the redis container is up and there's no check if the data was loaded.

If I do this it's more or less always successfull.

**test: sleep 30; redis-cli ping** Since this is not a proper solution, can the test be improved?

Sometimes it's also ok to do it two steps:

docker compose up redis -d
docker compose up -d

Event ID

No response

klemen-df avatar Oct 07 '24 06:10 klemen-df

Thanks for writing in. Is this error crashing your web instance?

hubertdeng123 avatar Oct 08 '24 22:10 hubertdeng123

It sometimes causes issues when starting, yes. It's a race condition.

Either way - it's not ok to say that if 'ping' goes throught, the redis service is ready. 'ping' just means that the container is up but it doesn't necessarily mean that the database is loaded.

klemen-df avatar Oct 09 '24 07:10 klemen-df

@klemen-df i did facing this issue. But in my case the sentry is not health from the start. So my workaround is to start the redis and flush the redis with assumption some cache or data is outdated and not well processed so it's safe for me to start flushing it up. Then everythings work fine

bijancot avatar Oct 12 '24 05:10 bijancot

what about this?

redis-cli INFO Persistence | grep loading

1: in loading 0: loaded

klemen-df avatar Oct 14 '24 08:10 klemen-df

what about this?

redis-cli INFO Persistence | grep loading

1: in loading 0: loaded

At that time, that's not crossing my mind (sorry). And i'm not checking that one

bijancot avatar Oct 14 '24 08:10 bijancot

Looks like what we can do is redis-cli ping | grep PONG

https://stackoverflow.com/questions/67904609/how-do-you-perform-a-healthcheck-in-the-redis-docker-image

hubertdeng123 avatar Nov 11 '24 22:11 hubertdeng123