srclient icon indicating copy to clipboard operation
srclient copied to clipboard

Schema not found errors when creating schemas

Open adiglp opened this issue 1 year ago • 3 comments
trafficstars

Hello,

First of all, thank you for this amazing module, it really helped us

We are using it for automating schema registry operations for other teams with Git. Sometimes we've found strange errors when creating schemas, example: "{"error_code":40403,"message":"Schema 2213 not found"}"

Our Schema Registry architecture is composed on multiple instances for high availability purposes. Checking module code, we realized "CreateSchema" method performs 2 operations:

  • Http requst to POST the new schema (https://github.com/riferrei/srclient/blob/master/schemaRegistryClient.go#L394)
  • "GetSchema" call using ID returned (https://github.com/riferrei/srclient/blob/master/schemaRegistryClient.go#L405)

Our hypothesis is, as the http requests are distributed between the different backends:

  1. POST is routed to backend 1 (could be master or not, will redirect to master if not), successfully registering the schema
  2. GET is routed to backend 2 (can't be master), which didn't fetch yet the change from the topic
  3. Schema not found error is returned
  4. Schema is available as it was successfully registered

At least this makes sense to us. I'm super happy contributing and making code changes, but i'd like to align with you on the approach, we've some things in mind:

  • New method similar to SetTimeout, example "SetCreateGetDelay". If set, it adds a delay between operations (https://github.com/riferrei/srclient/blob/master/schemaRegistryClient.go#L404)
  • New method called "SkipCreateGetSchema". If set, it skips getting the schema, decoupling it
  • Remove / decouple GetSchema from CreateSchema. For us make sense, probably for other usages can impact
  • Any other option proposal

If you need any additional detail, please let me know

Thanks in advance, Guille

adiglp avatar Apr 17 '24 11:04 adiglp

Hello, any news on that? We are facing the same issue.

javishere avatar Jul 04 '24 10:07 javishere

Hello @riferrei have you had the chance of checking on to this issue? It's also affecting us.

juanlucruz avatar Aug 19 '24 14:08 juanlucruz

Hi @adiglp, thank you for the thread and sorry for the late reply. I think decoupling them makes most sense. Could you provide a PR, preferably with a particular test please?

I'd also like to ask people facing the same issue, if this change is preferred or necessary (i.e. could you define a check on the caller function, and compare the error and do a getschema request there instead)

AtakanColak avatar Dec 15 '24 21:12 AtakanColak