karapace
karapace copied to clipboard
Published
20 hours ago •
Aiven-Open
Aiven-Open
Schema reader crashes when replaying protobuf
What happened?
Adding a protobuf schema causes an exception and schema registry becomes unresponsive.
What did you expect to happen?
Schema to be added and replay to continue. API to continue to be available.
What else do we need to know?
Tried with 3.14.0 and 3.15.0
Schema appears to be added OK, but replay fails in the same way, restarting the container results in the below error:
Starting Karapace Schema Registry
Cannot enable Sentry.io sending: importing 'sentry_sdk' failed
karapace MainThread INFO Karapace initialized
root MainThread INFO
Starting karapace schema registry
karapace.instrumentation.prometheus MainThread INFO Setting up prometheus metrics
karapace.coordinator.schema_coordinator MainThread INFO Metadata for topic has changed from [] to [Assignment(member_id='v0', metadata=b'{"version":2,"karapace_version":"3.15.0","host":"karapace-schema-registry","port":8081,"scheme":"http","master_eligibility":true}')].
karapace.coordinator.schema_coordinator MainThread INFO Discovered coordinator 0 for group karapace-schema-registry
karapace.coordinator.schema_coordinator MainThread INFO (Re-)joining group karapace-schema-registry
karapace.coordinator.schema_coordinator MainThread INFO Joined group 'karapace-schema-registry' (generation 16) with member_id karapace-schema-registry-73c87c16-78ce-4997-b000-456cd1111889
karapace.coordinator.schema_coordinator MainThread INFO Joined as follower.
karapace.coordinator.schema_coordinator MainThread INFO Successfully synced group karapace-schema-registry with generation 16
karapace.coordinator.schema_coordinator MainThread INFO Join complete, generation 16, member_id: 'karapace-schema-registry-73c87c16-78ce-4997-b000-456cd1111889', protocol: 'v0', member_assignment_bytes: b'{"master":"karapace-schema-registry-ad1354cc-1653-4808-8b57-6aeb6b1ec925","master_identity":{"version":2,"karapace_version":"3.14.0","host":"karapace-schema-registry","port":8081,"scheme":"http","master_eligibility":true},"error":0}'
karapace.kafka.consumer.KafkaConsumer schema-reader INFO Subscribing to topics ['_schemas'] and patterns []
karapace.schema_reader schema-reader INFO [Schema Topic] Creating '_schemas'
karapace.kafka.admin.KafkaAdminClient schema-reader INFO Creating new topic NewTopic(topic=_schemas,num_partitions=1) with replication factor 1
karapace.schema_reader schema-reader WARNING [Schema Topic] Already exists '_schemas'
aiohttp.access MainThread INFO 0.000669s - "GET /_health HTTP/1.1" 200 "kube-probe/1.26" response=570b request_body=-b
aiohttp.access MainThread INFO 0.001064s - "GET /_health HTTP/1.1" 200 "kube-probe/1.26" response=570b request_body=-b
karapace.schema_reader schema-reader INFO Replay progress (3.27): -1/0 (0 %) (recs/s -0.30561052807405376)
karapace.schema_reader schema-reader INFO Replay progress (0.2): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.2): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.2): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.2): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.2): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.2): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.2): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.2): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.2): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.2): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.2): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.2): -1/0 (0 %) (recs/s 0.0)
karapace.coordinator.schema_coordinator MainThread WARNING Heartbeat failed for group karapace-schema-registry because it is rebalancing
karapace.coordinator.schema_coordinator MainThread INFO (Re-)joining group karapace-schema-registry
karapace.coordinator.schema_coordinator MainThread INFO Joined group 'karapace-schema-registry' (generation 17) with member_id karapace-schema-registry-73c87c16-78ce-4997-b000-456cd1111889
karapace.coordinator.schema_coordinator MainThread INFO Elected group leader -- performing partition assignments using v0
karapace.coordinator.schema_coordinator MainThread INFO Creating assignment: 'karapace-schema-registry-73c87c16-78ce-4997-b000-456cd1111889', protocol: 'v0', members: [JoinGroupMemberData(member_id='karapace-schema-registry-73c87c16-78ce-4997-b000-456cd1111889', member_data=b'{"version":2,"karapace_version":"3.15.0","host":"karapace-schema-registry","port":8081,"scheme":"http","master_eligibility":true}')]
karapace.coordinator.schema_coordinator MainThread INFO Chose: 'karapace-schema-registry-73c87c16-78ce-4997-b000-456cd1111889' with url: 'http://karapace-schema-registry:8081' as the master
karapace.coordinator.schema_coordinator MainThread INFO Successfully synced group karapace-schema-registry with generation 17
karapace.coordinator.schema_coordinator MainThread INFO Join complete, generation 17, member_id: 'karapace-schema-registry-73c87c16-78ce-4997-b000-456cd1111889', protocol: 'v0', member_assignment_bytes: b'{"master":"karapace-schema-registry-73c87c16-78ce-4997-b000-456cd1111889","master_identity":{"version":2,"karapace_version":"3.15.0","host":"karapace-schema-registry","port":8081,"scheme":"http","master_eligibility":true},"error":0}'
karapace.schema_reader schema-reader INFO Replay progress (0.2): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.2): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.23): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader ERROR Unexpected exception in schema reader loop
Traceback (most recent call last):
File "/venv/lib/python3.10/site-packages/karapace/protobuf/schema.py", line 267, in __init__
self.proto_file_element = deserialize(schema)
File "/venv/lib/python3.10/site-packages/karapace/protobuf/serialization.py", line 157, in deserialize
serialized_pb = base64.b64decode(schema_b64, validate=True)
File "/usr/local/lib/python3.10/base64.py", line 86, in b64decode
raise binascii.Error('Non-base64 digit found')
binascii.Error: Non-base64 digit found
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/venv/lib/python3.10/site-packages/karapace/schema_reader.py", line 244, in run
self.handle_messages()
File "/venv/lib/python3.10/site-packages/karapace/schema_reader.py", line 390, in handle_messages
self.handle_msg(key, value)
File "/venv/lib/python3.10/site-packages/karapace/schema_reader.py", line 586, in handle_msg
self._handle_msg_schema(key, value)
File "/venv/lib/python3.10/site-packages/karapace/schema_reader.py", line 539, in _handle_msg_schema
parsed_schema = parse_protobuf_schema_definition(
File "/venv/lib/python3.10/site-packages/karapace/schema_models.py", line 75, in parse_protobuf_schema_definition
ProtobufSchema(schema_definition, references, dependencies)
File "/venv/lib/python3.10/site-packages/karapace/protobuf/schema.py", line 269, in __init__
self.proto_file_element = ProtoParser.parse(DEFAULT_LOCATION, schema)
File "/venv/lib/python3.10/site-packages/karapace/protobuf/proto_parser.py", line 629, in parse
return proto_parser.read_proto_file()
File "/venv/lib/python3.10/site-packages/karapace/protobuf/proto_parser.py", line 104, in read_proto_file
declaration = self.read_declaration(documentation, Context.FILE)
File "/venv/lib/python3.10/site-packages/karapace/protobuf/proto_parser.py", line 205, in read_declaration
result = self.read_message(location, documentation)
File "/venv/lib/python3.10/site-packages/karapace/protobuf/proto_parser.py", line 245, in read_message
declared = self.read_declaration(nested_documentation, Context.MESSAGE)
File "/venv/lib/python3.10/site-packages/karapace/protobuf/proto_parser.py", line 215, in read_declaration
result = self.read_one_of(documentation)
File "/venv/lib/python3.10/site-packages/karapace/protobuf/proto_parser.py", line 453, in read_one_of
fields.append(self.read_field_with_label(location, nested_documentation, None, atype))
File "/venv/lib/python3.10/site-packages/karapace/protobuf/proto_parser.py", line 389, in read_field_with_label
self.reader.require("=")
File "/venv/lib/python3.10/site-packages/karapace/protobuf/syntax_reader.py", line 35, in require
self.expect(self.read_char() == c, f"expected '{c}'")
File "/venv/lib/python3.10/site-packages/karapace/protobuf/syntax_reader.py", line 361, in expect
self.unexpected(message, location)
File "/venv/lib/python3.10/site-packages/karapace/protobuf/syntax_reader.py", line 370, in unexpected
raise IllegalStateException(f"Syntax error in {str(location)}: {message}")
karapace.protobuf.exception.IllegalStateException: Syntax error in :14:54: expected '='
karapace.schema_reader schema-reader INFO Replay progress (0.25): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.21): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.22): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.21): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.21): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.2): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.21): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.21): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.22): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.21): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.2): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.2): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.2): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.21): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.21): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.21): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.21): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.21): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.21): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.21): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.2): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.23): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.21): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.2): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.22): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.25): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.21): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.2): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.2): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.21): -1/0 (0 %) (recs/s 0.0)
karapace.schema_reader schema-reader INFO Replay progress (0.21): -1/0 (0 %) (recs/s 0.0)
This log seems to continue repeating forever, with various similar values in replay progress.
