pulsar-flink icon indicating copy to clipboard operation
pulsar-flink copied to clipboard

[QUESTION] Disable flink schema upload operation

Open nikolasten opened this issue 2 years ago • 2 comments

We are running a flink sink on topic that it does not have any pulsar schema. Basically we are publishing just json strings. If we dont add service account as admin role on tenant, flink fails with :

java.lang.RuntimeException: Failed to get schema information for persistent://production/backoffice/internal.notification-bet-activity
    at org.apache.flink.streaming.connectors.pulsar.internal.SchemaUtils.uploadPulsarSchema(SchemaUtils.java:65)
    at org.apache.flink.streaming.connectors.pulsar.FlinkPulsarSinkBase.uploadSchema(FlinkPulsarSinkBase.java:302)
    at org.apache.flink.streaming.connectors.pulsar.FlinkPulsarSinkBase.open(FlinkPulsarSinkBase.java:267)
    at org.apache.flink.streaming.connectors.pulsar.FlinkPulsarSink.open(FlinkPulsarSink.java:41)
    at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:34)
    at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
    at org.apache.flink.streaming.api.operators.StreamSink.open(StreamSink.java:46)
    at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:442)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:585)
    at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(StreamTask.java:565)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:540)
    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:759)
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.pulsar.client.admin.PulsarAdminException$NotAuthorizedException: Don't have permission to administrate resources on this tenant

Is it possible to disable schema uploads done by flink connector? If not, what is preferred way to add this permission and not to add full admin access on tenant?

nikolasten avatar Apr 14 '22 08:04 nikolasten

I see that upload schema call in releases after 1.13.1.4 is removed https://github.com/streamnative/pulsar-flink/blob/release-1.13.1.4/pulsar-flink-connector/src/main/java/org/apache/flink/streaming/connectors/pulsar/FlinkPulsarSinkBase.java#L267 When will the stable version of 1.13.1.5 is going to be released? At the moment there are only release candidate versions

nikolasten avatar Apr 18 '22 10:04 nikolasten

@imaffe Could you help release a stable 1.13 once the pipeline is finished.

nlu90 avatar Apr 27 '22 22:04 nlu90