Concurrent POSTs to create schema or new schema versions are failing with 500

[burandobata]

Hey,

it looks like when more than one SR client try to register a new schema at the same time, all but one of them will get general, non-retriable HTTP 500 error. Similar situation happens when more than one client try to register a new version of existing schema, however an error is slightly different. This situation is easily reproducible with a simple piece of code which can do this using 2 threads only.

Under the hood it looks like it creates couple of db transactions that want to create the same entry. Only 1 can pass in this situation as the rest will violate unique constraints.

DB errors from new schema creation:

ERROR:  could not serialize access due to concurrent update
STATEMENT:  INSERT INTO "schema_metadata_info" ("name", "description", "schemaGroup", "type", "compatibility", "validationLevel", "timestamp", "evolve") VALUES($1,$2,$3,$4,$5,$6,$7,$8) ON CONFLICT ON CONSTRAINT schema_metadata_info_pkey DO UPDATE SET "name" = $9, "description" = $10, "schemaGroup" = $11, "type" = $12, "compatibility" = $13, "validationLevel" = $14, "timestamp" = $15, "evolve" = $16
        RETURNING *
ERROR:  current transaction is aborted, commands ignored until end of transaction block
STATEMENT:  INSERT INTO "schema_metadata_info" ("name", "description", "schemaGroup", "type", "compatibility", "validationLevel", "timestamp", "evolve") VALUES($1,$2,$3,$4,$5,$6,$7,$8) ON CONFLICT ON CONSTRAINT schema_metadata_info_pkey DO UPDATE SET "name" = $9, "description" = $10, "schemaGroup" = $11, "type" = $12, "compatibility" = $13, "validationLevel" = $14, "timestamp" = $15, "evolve" = $16

DB errors from new version creation:

ERROR:  current transaction is aborted, commands ignored until end of transaction block
STATEMENT:  SELECT * FROM "schema_metadata_info" WHERE "name" = $1
ERROR:  duplicate key value violates unique constraint "schema_version_info_pkey"
DETAIL:  Key (name, version)=(test_topic4, 3) already exists.
STATEMENT:  INSERT INTO "schema_version_info" ("schemaMetadataId", "schemaText", "name", "description", "version", "timestamp", "fingerprint", "state") VALUES( $1,$2,$3,$4,$5,$6,$7,$8)
        RETURNING *

Can this be looked at?

I believe this is the same problem that has caused this: https://github.com/hortonworks/registry/issues/430

thanks, Marek

[jbrackett]

I'm seeing something similar as well. Mine is caused by a deadlock. Not sure if it's the same issue or just similar but manifests itself most prominently when running in a limited environment like a docker container

! Causing: com.hortonworks.registries.storage.exception.StorageException: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction ! at com.hortonworks.registries.storage.impl.jdbc.provider.sql.factory.AbstractQueryExecutor$QueryExecution.executeUpdateWithReturningGeneratedKey(AbstractQueryExecutor.java:378) ! at

[burandobata]

Hey,

this is solved by: https://github.com/hortonworks/registry/pull/518 Can this be added to new release?

[satishd]

@burandobata This is already on master and it will be part of next release.

[burandobata]

@satishd Thanks for info and when are you planning to have a next release?

[satishd]

@burandobata Next release is not yet planned. I will update through google groups when it is planned.

[gcsaba2]

Done