function-mesh
function-mesh copied to clipboard
[Bug] function mesh is not friendly to batch source configuration.
Version
- Pulsar: 2.10.0.7
- JDK: 1.8
Minimal reproduce step
Run a batch source connector with function mesh, and the CRD configuration is as follows:
apiVersion: compute.functionmesh.io/v1alpha1
kind: Source
metadata:
name: google-bigquery-source-sample
spec:
image: danpi1hao/pulsar-io-bigquery:2.10.0.7
className: org.apache.pulsar.functions.source.batch.BatchSourceExecutor
batchSourceConfig:
discoveryTriggererClassName: org.apache.pulsar.ecosystem.io.bigquery.source.BigQueryOnceTrigger
replicas: 1
sourceConfig:
projectId: SECRET
datasetName: convert
tableName: demo01
pulsar:
pulsarConfig: "test-pulsar-source-config"
java:
jar: connectors/pulsar-io-bigquery-2.10.0.7.jar
clusterName: pulsar
What did you expect to see?
The batch source connector can work well on function mesh.
What did you see instead?
It seems that function mesh does not provide good support for mapping from batchSourceConfig
to sourceConfig
. So I am currently manually mapping the batchSourceConfig
to the sourceConfig
, as shown below:
apiVersion: compute.functionmesh.io/v1alpha1
kind: Source
metadata:
name: google-bigquery-source-sample
spec:
image: danpi1hao/pulsar-io-bigquery:2.10.0.7
className: org.apache.pulsar.functions.source.batch.BatchSourceExecutor
replicas: 1
sourceConfig:
__BATCHSOURCECLASSNAME__: org.apache.pulsar.ecosystem.io.bigquery.BigQuerySource
__BATCHSOURCECONFIGS__:
'{"discoveryTriggererClassName":"org.apache.pulsar.ecosystem.io.bigquery.source.BigQueryOnceTrigger"}'
projectId: SECRET
datasetName: convert
tableName: demo01
pulsar:
pulsarConfig: "test-pulsar-source-config"
java:
jar: connectors/pulsar-io-bigquery-2.10.0.7.jar
clusterName: pulsar