pranadb icon indicating copy to clipboard operation
pranadb copied to clipboard

protolib selector library should validate that the selector matches the schema

Open mightyguava opened this issue 3 years ago • 0 comments

That way you can get an error when the source is created. Otherwise you'll only see ingestion failures in the logs after the source starts consuming.

Right now if you deploy a source with an invalid selector, e.g.

create source tx_pending(
  transaction_id varchar,
  customer_token varchar,
  payment_type varchar,
  primary key (transaction_id)
) with (
  brokername = "testbroker",
  topicname = "my_events",
  headerencoding = "stringbytes",
  keyencoding = "stringbytes",
  valueencoding = "protobuf:squareup.cash.foo.bar",
  columnselectors = (
    transaction.token,
    customer_token,
    transaction.foobar,
  )
);

I.e. foobar is not a field on the proto.

Then we don't get an error at source create time, the error occurs the first time a proto is decoded from Kafka.

Instead we should pro-actively check the selectors at source create time and return errors if they are invalid.

mightyguava avatar Sep 23 '21 20:09 mightyguava