loggie icon indicating copy to clipboard operation
loggie copied to clipboard

feat(source): support elasticsearch

Open 5idu opened this issue 3 years ago • 1 comments

Proposed Changes:

  • add elasticsearch source

Which issue(s) this PR fixes:

Fixes #198

Additional documentation:

Support pull data from elastic regularly,the configuration is as follows:

pipelines:
  - name: local
    sources:
      - type: elasticsearch
        name: elastic
        hosts:
          - "localhost:9200"
          - "localhost:9201"
        username: "bob"
        password: "bob"
        schema: ""
        sniff: false
        gzip: true
        includeFields: # pull selected field
          - Title
          - Content
          - Author
        excludeFields: # exclude selected field
          - Content
        query: | # elastic query phrases
          {
            "match": {"Title": "bob"}
          }
        index: "blog*"
        size: 10 # data size per fetch
        interval: 30s # pull data frequency
        timeout: 5s # pull timeout
        db: 
          flushTimeout: 2s # persistent the elastic pull location frequency
          cleanInactiveTimeout: 24h # delete the db record after the time
          cleanScanInterval: 1h # check the expired db record frequency
        
    sink:
      type: dev
      printEvents: true
      codec:
        pretty: true
        type: json
        beatsFormat: true

5idu avatar Aug 26 '22 01:08 5idu

LGTM, we will test it later.

ethfoo avatar Sep 04 '22 14:09 ethfoo