self-hosted
self-hosted copied to clipboard
getsentry
ERROR: duplicate key value violates unique constraint
Self-Hosted Version
23.5.1
CPU Architecture
x86_64
Docker Version
24.0.2
Docker Compose Version
2.18.1
Steps to Reproduce
- Upgrade Sentry from pre-postgres-14 version (i.e. 23.3.0) to latest version
- Make sure there is a moderate amount of incoming events, and issues are being created
- Check postgres container logs
Expected Result
There should be no errors.
Actual Result
A lot of errors like this:
ERROR: duplicate key value violates unique constraint "sentry_environmentprojec_project_id_environment_i_91da82f2_uniq"
DETAIL: Key (project_id, environment_id)=(175, 1) already exists.
STATEMENT: INSERT INTO "sentry_environmentproject" ("project_id", "environment_id", "is_hidden") VALUES (175, 1, NULL) RETURNING "sentry_environmentproject"."id"
ERROR: duplicate key value violates unique constraint "sentry_grouprelease_group_id_release_id_envi_044354c8_uniq"
DETAIL: Key (group_id, release_id, environment)=(5040920, 17693, production) already exists.
STATEMENT: INSERT INTO "sentry_grouprelease" ("project_id", "group_id", "release_id", "environment", "first_seen", "last_seen") VALUES (38, 5040920, 17693, 'production', '2023-06-01T11:40:07.927899+00:00'::timestamptz, '2023-06-01T11:40:07.927899+00:00'::timestamptz) RETURNING "sentry_grouprelease"."id"
Event ID
No response
Wondering if your sequence is out of sync. Could you see if that is the case using the steps here? https://stackoverflow.com/questions/4448340/postgresql-duplicate-key-violates-unique-constraint
Unfortunately, I am not a database expert. Can you please help me figure out which commands I need to execute to verify and fix the issue? Please note that I never tinkered with Sentry's postgres, so if something went out of sync, that's definitely not a result of manual intervention.
Here's what I see:
postgres=# select max(id) from sentry_environmentproject;
max
--------
792802
(1 row)
postgres=# SELECT nextval('sentry_environmentproject_id_seq');
nextval
---------
799322
(1 row)
Similar issue here. Happens after upgrade from 23.3.1 to 23.4.0 with the upgrade to postgresql. I did have an error during the installation because of duplicate entries in sentry_releaseprojectenvironment which I had to remove manually to get overcome a duplicate key error during reindex.
installation log
Re-indexing database: sentry
ERROR: could not create unique index "sentry_releaseprojectenvironmen_project_id_d2be2f28c78caf7_uniq"
DETAIL: Key (project_id, release_id, environment_id)=(11, 5198, 2) is duplicated.
Error in install/upgrade-postgres.sh:38.
'docker exec sentry-self-hosted-postgres-1 psql -qAt -U postgres -d ${db} -c "reindex database ${db};"' exited with status 1
-> install.sh:main:33
--> install/upgrade-postgres.sh:source:38
2023-06-06 15:53:15.444 UTC [15601] STATEMENT: INSERT INTO "sentry_environmentproject" ("project_id", "environment_id", "is_hidden") VALUES (15, 2, NULL) RETURNING "sentry_environmentproject"."id"
2023-06-06 15:53:35.614 UTC [15705] ERROR: duplicate key value violates unique constraint "sentry_organizationonboar_organization_id_47e98e05cae29cf3_uniq"
2023-06-06 15:53:35.614 UTC [15705] DETAIL: Key (organization_id, task)=(2, 5) already exists.
2023-06-06 15:53:35.614 UTC [15705] STATEMENT: INSERT INTO "sentry_organizationonboardingtask" ("organization_id", "user_id", "status", "completion_seen", "date_completed", "project_id", "data", "task") VALUES (2, NULL, 1, NULL, '2023-06-06T15:53:35.611610+00:00'::timestamptz, 13, '{}', 5) RETURNING "sentry_organizationonboardingtask"."id"
2023-06-06 15:53:35.622 UTC [15706] ERROR: duplicate key value violates unique constraint "sentry_organizationonboar_organization_id_47e98e05cae29cf3_uniq"
2023-06-06 15:53:35.622 UTC [15706] DETAIL: Key (organization_id, task)=(2, 5) already exists.
2023-06-06 15:53:35.622 UTC [15706] STATEMENT: INSERT INTO "sentry_organizationonboardingtask" ("organization_id", "user_id", "status", "completion_seen", "date_completed", "project_id", "data", "task") VALUES (2, NULL, 1, NULL, '2023-06-06T15:53:35.619722+00:00'::timestamptz, 12, '{}', 5) RETURNING "sentry_organizationonboardingtask"."id"
2023-06-06 15:53:40.217 UTC [15748] ERROR: duplicate key value violates unique constraint "sentry_environmentproject_project_id_29250c1307d3722b_uniq"
2023-06-06 15:53:40.217 UTC [15748] DETAIL: Key (project_id, environment_id)=(11, 2) already exists.
2023-06-06 15:53:40.217 UTC [15748] STATEMENT: INSERT INTO "sentry_environmentproject" ("project_id", "environment_id", "is_hidden") VALUES (11, 2, NULL) RETURNING "sentry_environmentproject"."id"
2023-06-06 15:53:40.225 UTC [15749] ERROR: duplicate key value violates unique constraint "sentry_environmentproject_project_id_29250c1307d3722b_uniq"
2023-06-06 15:53:40.225 UTC [15749] DETAIL: Key (project_id, environment_id)=(11, 2) already exists.
2023-06-06 15:53:40.225 UTC [15749] STATEMENT: INSERT INTO "sentry_environmentproject" ("project_id", "environment_id", "is_hidden") VALUES (11, 2, NULL) RETURNING "sentry_environmentproject"."id"
2023-06-06 15:54:03.426 UTC [15898] ERROR: duplicate key value violates unique constraint "sentry_environmentproject_project_id_29250c1307d3722b_uniq"
2023-06-06 15:54:03.426 UTC [15898] DETAIL: Key (project_id, environment_id)=(5, 2) already exists.
2023-06-06 15:54:03.426 UTC [15898] STATEMENT: INSERT INTO "sentry_environmentproject" ("project_id", "environment_id", "is_hidden") VALUES (5, 2, NULL) RETURNING "sentry_environmentproject"."id"
2023-06-06 15:54:37.333 UTC [16129] ERROR: duplicate key value violates unique constraint "sentry_environmentproject_project_id_29250c1307d3722b_uniq"
2023-06-06 15:54:37.333 UTC [16129] DETAIL: Key (project_id, environment_id)=(12, 5) already exists.
2023-06-06 15:54:37.333 UTC [16129] STATEMENT: INSERT INTO "sentry_environmentproject" ("project_id", "environment_id", "is_hidden") VALUES (12, 5, NULL) RETURNING "sentry_environmentproject"."id"
sentry=# select max(id) from sentry_environmentproject;
max
-------
25791
(1 row)
sentry=# SELECT nextval('sentry_environmentproject_id_seq');
nextval
---------
75872
(1 row)
@AlexanderRydberg I don't have the exact commands anymore, but basically it was
- pg_dump sentry_releaseprojectenvironment > dump.sql
- remove duplicate entries as shown in the output from (re)index
- cat dump.sql | pgsql
- repeat from step 2 until all duplicate entries are resolved
all duplicate entries were from 2 months ago when we had issues with our server and the table seems to have gotten corrupted. i don't think this is related with the duplicate key errors which are happening after the upgrade
ouch scary :fearful: We have this "issue" on both test and prod-env. Hopefully there will come a official fix for this, I guess that a lot more users have this error.
Dont know how this effects Sentry tho
I'd like to highlight that @roock most probably having a different issue. I didn't have any errors during the upgrade. Everything went smoothly, no errors, no warnings. I didn't run any postgres commands manually or interfered with postgres in any other way.
@afaianswq I didnt have any errors dring the upgrade either nor any postgres changes manually
@afaianswq Yeah, I most likely had a different (and probably unique) issue during upgrade. After the upgrade is completed, I am seeing the same errors in the database log as you are seeing
@AlexanderRydberg @afaianswq Are you missing transactions and or issues inside Sentry when the database errors are happening or is it "just" errors in the database log?
This issue has gone three weeks without activity. In another week, I will close it.
But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!
"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This is a serious issue and it hasn't been fixed. Please reopen and set appropriate status so it won't automatically close again.
This is a serious issue and it hasn't been fixed. Please reopen and set appropriate status so it won't automatically close again.
Are there any errors that are happening? It is actually pretty normal for the duplicate key value errors to come up in postgres container logs. It has to do something with implementation details when we were using an older version of postgres that didn't support UPSERT.
See here for more details: https://github.com/getsentry/sentry/issues/8004#issuecomment-613123709
Otherwise, if your self-hosted Sentry instance appears to be working as intended, this shouldn't be something to worry about.
This issue has gone three weeks without activity. In another week, I will close it.
But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!
"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀
![getsantry[bot] avatar](https://avatars.githubusercontent.com/in/66573?v=4 "Published by a pub.dev verified publisher"){kind=small}
We're getting this error which is preventing most of self-hosted from working. We tried to do an upgrade that failed because it didn't like some table id on a 22.x release and decided to backup and restore on a clean install but now that clean install has most pages not loading (getting 404s) and constant spamming of the pgsql errors.
Worth noting restore says "No sequences found" multiple times with no further explanation in logs.
Did you do a backup/restore on the same version of self-hosted? It would be helpful to know the exact versions that you were upgrading to/from. There may have been a failed database migration
Did you do a backup/restore on the same version of self-hosted? It would be helpful to know the exact versions that you were upgrading to/from. There may have been a failed database migration
As stated, there was an error and I wasn't paying close enough attention to note the exact version because documentation doesn't list any caveats after 21.x and we were on like 22.7.x and upgraded to 23.7.0 which errored out on the migration and restoring an old backup didn't work either.
Ah yes, some people have reported that 23.7.0 have created issues due to the way some migrations were squashed and the django 3 upgrade in Sentry. We've added documentation to tell users to upgrade to 23.6.2 before moving to versions higher: https://develop.sentry.dev/self-hosted/releases/
Ah yes, some people have reported that 23.7.0 have created issues due to the way some migrations were squashed and the django 3 upgrade in Sentry. We've added documentation to tell users to upgrade to 23.6.2 before moving to versions higher: https://develop.sentry.dev/self-hosted/releases/
Would have been great to know, like, before we upgraded, but thanks 😜
We have the same issue. Unable to update from 23.3.1 to 23.4.0 or even 23.5.0.
ERROR: could not create unique index "sentry_environmentproject_project_id_29250c1307d3722b_uniq"
DETAIL: Key (project_id, environment_id)=(55, 11) is duplicated.
Error in install/upgrade-postgres.sh:38.
We have the same issue. Unable to update from 23.3.1 to 23.4.0 or even 23.5.0.
Did you attempt to upgrade to 23.7.0+ first? Might be why this issue is appearing for you
Regarding https://develop.sentry.dev/self-hosted/releases/#hard-stops I should upgrade first to 23.6.2 if I'm upgrading prior this version. I did try to upgrade to 23.6.2 but with no luck, same error about the duplicated key.
Regarding https://develop.sentry.dev/self-hosted/releases/#hard-stops I should upgrade first to 23.6.2 if I'm upgrading prior this version. I did try to upgrade to 23.6.2 but with no luck, same error about the duplicated key.
There may now have been corrupted data in your db, sorry that you're experiencing that. You may need to attempt a clean install to fix your setup.