bitsail icon indicating copy to clipboard operation
bitsail copied to clipboard

[BitSail][Connector] Support Pulsar source connector

Open lichang-bd opened this issue 3 years ago • 4 comments

Is your feature request related to a problem? Please describe

A clear and concise description of what the problem is.

Example: I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

lichang-bd avatar Nov 08 '22 17:11 lichang-bd

I'd like to take this issue.

qidian99 avatar Nov 17 '22 18:11 qidian99

A sample job config is as follows:

{
  "job": {
    "common": {
      "job_type": "STREAMING",
      "job_plugin_lib_dir": "plugin",
      "job_plugin_conf_dir": "plugin_conf",
      "enable_dynamic_loader": true,
      "instance_id": "1",
      "internal_instance_id": "1",
      "extra_properties": {
        "update-mode": "append"
      }
    },
    "reader": {
      "pulsar": {
        "source": {
          "enableAutoAcknowledgeMessage": true
        },
        "client": {
          "serviceUrl": "pulsar://lcoalhost:8080"
        },
        "admin": {
          "adminUrl": "http://localhost:6650",
          "topics": "TBW102",
          "topicMode": "list"
        },
        "consumer": {
          "startCursorMode": "latest",
          "subscriptionType": "Exclusive"
        }
      },

      "columns": [
        {
          "name": "ID",
          "type": "string"
        }
      ],
      "format_type": "json",
      "class": "com.bytedance.bitsail.connector.pulsar.source.PulsarSourceV1"
    },
    "writer": {
      "class": "com.bytedance.bitsail.connector.legacy.print.sink.PrintSink"
    }
  }
}

sample output: image

P.S., I always get the slf4j warnings SLF4J: Class path contains multiple SLF4J bindings., SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] , and no log is printed which impacts the testing processing. Any help is appreciated.

qidian99 avatar Nov 18 '22 19:11 qidian99

@qidian99 assigned to you!

garyli1019 avatar Nov 22 '22 02:11 garyli1019

Reminder, is there any problem encountered in this task, we can discuss it together @qidian99

lichang-bd avatar Jan 16 '23 03:01 lichang-bd