Unable to access confluent compatible schema from apicurio ccompat v6

[kumud-hs]

Apicurio version: 2.2.3.Final Running on Docker environment using: apicurio/apicurio-registry-sql:2.2.3.Final

curl -X GET -H "Accept:application/json" http://hostname/apis/ccompat/v6/subjects/<subject-name>/versions/latest

This gives 400 with response {"message":"Failed to discover artifact type from content.","error_code":0} and in the logs I can see [Fatal Error] :1:1: Content is not allowed in prolog.

This is happening only for a few schemas.

The same schema can be accessed using the below API for all my schemas.

curl -X GET -H "Accept:application/json" http://hostname/apis/ccompat/v6/schemas/ids/19

[kumud-hs]

@carlesarnal Any other information that you need regarding this issue?

[carlesarnal]

Hi @kumud-hs, yes, if you can share one of those schemas, that would be great.

[kumud-hs]

Hey @carlesarnal I am so sorry, I somehow lost the schema when I changed my source debezium payloads in order to workaround the issue. But, I can tell the case when it is happening if it helps to reproduce.

If you are aware of debezium, it is a nested payload that has both before and after fields with the same schema. In Apicurio, the after field's schema was referencing the schema name of the before field. That is when I saw the error in the apicurio logs. For non-nested payloads and without these self-referencing schemas, it was working fine.

Sample payload of debezium.

{
  "payload": {
    "before": { 
      "id": 1004,
      "first_name": "Anne",
      "last_name": "Kretchmar",
      "email": "[email protected]"
    },
    "after": { 
      "id": 1004,
      "first_name": "Anne Marie",
      "last_name": "Kretchmar",
      "email": "[email protected]"
    },
    "source": { 
      "version": "1.9.2.Final",
      "name": "mysql-server-1",
      "connector": "mysql",
      "name": "mysql-server-1",
      "ts_ms": 1465581029100,
      "snapshot": false,
      "db": "inventory",
      "table": "customers",
      "server_id": 223344,
      "gtid": null,
      "file": "mysql-bin.000003",
      "pos": 484,
      "row": 0,
      "thread": 7,
      "query": "UPDATE customers SET first_name='Anne Marie' WHERE id=1004"
    },
    "op": "u", 
    "ts_ms": 1465581029523 
  }
}

[pedromazala]

Is there an update on that? I'm having a similar issue. I can list schemas through http://$APICURIO/apis/ccompat/v6/schemas/ids/1 but when I call http://$APICURIO/apis/ccompat/v6/subjects I get an empty list. I was expecting to have the subjects mapped to the schemas.

Should I do something to map a schema to a subject?

[carlesarnal]

There are no updates since it's pretty difficult to reproduce the issue without the problematic schema. Nothing should be done to map a schema to a subject (if the subject does not exist, it will be created when the schema is created). If you provide more details (schema being used, schema type etc), I can try to reproduce the issue.

[pedromazala]

I'll check if I can send the schema and I'll let you know. Thank you really much for the reply @carlesarnal.

Anything else I'll send on this thread.

PS: my problem is with another schema. Just found the issue and was willing to know the outcome. Not trying to rush 😅

[sg1fan]

I have this issue using the debezium connector. I can retrieve every key schema through subjects/<namespace>-key/versions/latest but every value schema through subjects/<namespace>-value/versions/latest returns this error. However, I can access the value schema through schemas/ids/<id>. Here is a concrete example for a table named avatar_img:

curl -X GET http://localhost:8080/apis/ccompat/v6/schemas/ids/3

{"schema":"{\"type\":\"record\",\"name\":\"Envelope\",\"namespace\":\"pg_server.public.avatar_img\",\"fields\":[{\"name\":\"before\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"Value\",\"fields\":[{\"name\":\"id\",\"type\":{\"type\":\"string\",\"connect.version\":1,\"connect.default\":\"00000000-0000-0000-0000-000000000000\",\"connect.name\":\"io.debezium.data.Uuid\"},\"default\":\"00000000-0000-0000-0000-000000000000\"},{\"name\":\"url_lg\",\"type\":{\"type\":\"string\",\"connect.default\":\"\"},\"default\":\"\"},{\"name\":\"url_md\",\"type\":{\"type\":\"string\",\"connect.default\":\"\"},\"default\":\"\"},{\"name\":\"url_sm\",\"type\":{\"type\":\"string\",\"connect.default\":\"\"},\"default\":\"\"},{\"name\":\"created_at\",\"type\":{\"type\":\"string\",\"connect.version\":1,\"connect.default\":\"1970-01-01T00:00:00Z\",\"connect.name\":\"io.debezium.time.ZonedTimestamp\"},\"default\":\"1970-01-01T00:00:00Z\"}],\"connect.name\":\"pg_server.public.avatar_img.Value\"}],\"default\":null},{\"name\":\"after\",\"type\":[\"null\",\"Value\"],\"default\":null},{\"name\":\"source\",\"type\":\"io.debezium.connector.postgresql.Source\"},{\"name\":\"op\",\"type\":\"string\"},{\"name\":\"ts_ms\",\"type\":[\"null\",\"long\"],\"default\":null},{\"name\":\"transaction\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"ConnectDefault\",\"namespace\":\"io.confluent.connect.avro\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"total_order\",\"type\":\"long\"},{\"name\":\"data_collection_order\",\"type\":\"long\"}]}],\"default\":null}],\"connect.name\":\"pg_server.public.avatar_img.Envelope\"}","references":[{"name":"io.debezium.connector.postgresql.Source","subject":"io.debezium.connector.postgresql.Source","version":1}]}

Some debezium connector configurations pertaining to apicurio:

key.converter=io.apicurio.registry.utils.converter.AvroConverter
key.converter.apicurio.registry.url=http://localhost:8080/apis/registry/v2
key.converter.apicurio.registry.auto-register=true
key.converter.apicurio.registry.find-latest=true
value.converter=io.apicurio.registry.utils.converter.AvroConverter
value.converter.apicurio.registry.url=http://localhost:8080/apis/registry/v2
value.converter.apicurio.registry.auto-register=true
value.converter.apicurio.registry.find-latest=true

Using image apicurio-registry-mem:2.2.5.Final.

[carlesarnal]

I have found the issue. When the type is being discovered getting the schema, the references were not passed down and it was failing because of that. I have created a PR that solves this problem.