function-mesh icon indicating copy to clipboard operation
function-mesh copied to clipboard

[Bug] function mesh is not friendly to batch source configuration.

Open danpi opened this issue 2 years ago • 0 comments

Version

  • Pulsar: 2.10.0.7
  • JDK: 1.8

Minimal reproduce step

Run a batch source connector with function mesh, and the CRD configuration is as follows:

apiVersion: compute.functionmesh.io/v1alpha1  
kind: Source  
metadata:  
  name: google-bigquery-source-sample
spec:
  image: danpi1hao/pulsar-io-bigquery:2.10.0.7
  className: org.apache.pulsar.functions.source.batch.BatchSourceExecutor
  batchSourceConfig:
    discoveryTriggererClassName: org.apache.pulsar.ecosystem.io.bigquery.source.BigQueryOnceTrigger
  replicas: 1
  sourceConfig:
     projectId: SECRET
     datasetName: convert
     tableName: demo01
  pulsar:
    pulsarConfig: "test-pulsar-source-config"
  java:
    jar: connectors/pulsar-io-bigquery-2.10.0.7.jar
    clusterName: pulsar

What did you expect to see?

The batch source connector can work well on function mesh.

What did you see instead?

1666258931414

It seems that function mesh does not provide good support for mapping from batchSourceConfig to sourceConfig. So I am currently manually mapping the batchSourceConfig to the sourceConfig, as shown below:

apiVersion: compute.functionmesh.io/v1alpha1  
kind: Source  
metadata:  
  name: google-bigquery-source-sample
spec:
  image: danpi1hao/pulsar-io-bigquery:2.10.0.7
  className: org.apache.pulsar.functions.source.batch.BatchSourceExecutor
  replicas: 1
  sourceConfig:
     __BATCHSOURCECLASSNAME__: org.apache.pulsar.ecosystem.io.bigquery.BigQuerySource
     __BATCHSOURCECONFIGS__:
'{"discoveryTriggererClassName":"org.apache.pulsar.ecosystem.io.bigquery.source.BigQueryOnceTrigger"}'
     projectId: SECRET
     datasetName: convert
     tableName: demo01
  pulsar:
    pulsarConfig: "test-pulsar-source-config"
  java:
    jar: connectors/pulsar-io-bigquery-2.10.0.7.jar
    clusterName: pulsar

danpi avatar Oct 20 '22 13:10 danpi