loggie
loggie copied to clipboard
feat(source): support elasticsearch
Proposed Changes:
- add elasticsearch source
Which issue(s) this PR fixes:
Fixes #198
Additional documentation:
Support pull data from elastic regularly,the configuration is as follows:
pipelines:
- name: local
sources:
- type: elasticsearch
name: elastic
hosts:
- "localhost:9200"
- "localhost:9201"
username: "bob"
password: "bob"
schema: ""
sniff: false
gzip: true
includeFields: # pull selected field
- Title
- Content
- Author
excludeFields: # exclude selected field
- Content
query: | # elastic query phrases
{
"match": {"Title": "bob"}
}
index: "blog*"
size: 10 # data size per fetch
interval: 30s # pull data frequency
timeout: 5s # pull timeout
db:
flushTimeout: 2s # persistent the elastic pull location frequency
cleanInactiveTimeout: 24h # delete the db record after the time
cleanScanInterval: 1h # check the expired db record frequency
sink:
type: dev
printEvents: true
codec:
pretty: true
type: json
beatsFormat: true
LGTM, we will test it later.