fluent-bit-kubernetes-logging
fluent-bit-kubernetes-logging copied to clipboard
How to connect with elastic cloud?
I organized ek stack by elastic cloud. But i can't found any field for elastic cloud user and password. I tried this but i can't get any connections.
output-elasticsearch.conf: |
[OUTPUT]
Name es
Match *
Host ${FLUENT_ELASTICSEARCH_HOST}
Port ${FLUENT_ELASTICSEARCH_PORT}
HTTP_User ${FLUENT_ELASTICSEARCH_USER}
HTTP_Passwd ${FLUENT_ELASTICSEARCH_PASSWORD}
Logstash_Format On
Retry_Limit False
@xncbf : I have a similar issue. Do you solve yours ? Here is the fluent bit documentation about elasticsearch output plugin, it looks like you have the right parameters.
Here is my log output from a random fluent-bit 1.3.3 daemonset :
[2019/11/29 12:44:51] [error] [io fd=50] error sending data to: elasticsearch-es-http:9200 (Broken pipe)
[2019/11/29 12:44:51] [error] [src/flb_http_client.c:844 errno=25] Inappropriate ioctl for device
[2019/11/29 12:44:51] [ warn] [out_es] http_do=-1 URI=/_bulk
[2019/11/29 12:46:28] [error] [src/flb_http_client.c:844 errno=32] Broken pipe
[2019/11/29 12:46:28] [ warn] [out_es] http_do=-1 URI=/_bulk
[2019/11/29 12:51:40] [error] [io fd=50] error sending data to: elasticsearch-es-http:9200 (Broken pipe)
[2019/11/29 12:51:40] [error] [src/flb_http_client.c:844 errno=25] Inappropriate ioctl for device
[2019/11/29 12:51:40] [ warn] [out_es] http_do=-1 URI=/_bulk
And my conf :
output-elasticsearch.conf: |
[OUTPUT]
Name es
Match *
Host ${FLUENT_ELASTICSEARCH_HOST}
Port ${FLUENT_ELASTICSEARCH_PORT}
Logstash_Format On
Replace_Dots On
Retry_Limit False
HTTP_User ${FLUENT_ELASTICSEARCH_USER}
HTTP_Passwd ${FLUENT_ELASTICSEARCH_PASSWORD}
@ledroide , did you find out the solution?
I was having the same problem as you, which I fixed by enabling tls
on the output configuration.
output-elasticsearch.conf: |
[OUTPUT]
Name es
Match *
Host ${FLUENT_ELASTICSEARCH_HOST}
Port ${FLUENT_ELASTICSEARCH_PORT}
HTTP_User ${FLUENT_ELASTICSEARCH_USER}
HTTP_Passwd ${FLUENT_ELASTICSEARCH_PASSWORD}
Logstash_Format On
Replace_Dots On
Retry_Limit False
tls On
The errors stopped being logged, by I still couldn't make fluent-bit send anything to my Elasticsearch hosted on Elastic Cloud.
I'm having this output now on fluent-bit containers after adding tls:
Fluent Bit v1.3.11
Copyright (C) Treasure Data
[2020/04/23 18:52:17] [ info] [storage] version=1.0.3, initializing...
[2020/04/23 18:52:17] [ info] [storage] in-memory
[2020/04/23 18:52:17] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/04/23 18:52:17] [ info] [engine] started (pid=1)
[2020/04/23 18:52:17] [ info] [filter_kube] https=1 host=kubernetes.default.svc port=443
[2020/04/23 18:52:17] [ info] [filter_kube] local POD info OK
[2020/04/23 18:52:17] [ info] [filter_kube] testing connectivity with API server...
[2020/04/23 18:52:17] [ info] [filter_kube] API server connectivity OK
[2020/04/23 18:52:17] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2020/04/23 18:52:17] [ info] [sp] stream processor started
UPDATE
Actually the logs are working for me now.
The only thing left for me was to create the Index Pattern in Kibana, so I could visualize the logs. I created a Index Pattern for logstash-*
, and can see the logs now. But I don't why exactly they are being created in that index, since the default is fluentbit
.
But I also think @ledroide 's problem is not http, as I said before. I tested and it also worked.
Besides using tls On
make sure not to include the protocol (https://) part in your FLUENT_ELASTICSEARCH_HOST. At least that was my issue.
Change the following to solve the problem
fluent-bit-configmap.yaml
output-elasticsearch.conf: | [OUTPUT] Name es Match * Host ${FLUENT_ELASTICSEARCH_HOST} Port ${FLUENT_ELASTICSEARCH_PORT} HTTP_User ${FLUENT_ELASTICSEARCH_USER} HTTP_Passwd ${FLUENT_ELASTICSEARCH_PASSWORD} Logstash_Format On Logstash_Prefix k8s-fluentbit Replace_Dots On Retry_Limit False tls On tls.verify Off Type flb
fluent-bit-ds.yaml env: - name: FLUENT_ELASTICSEARCH_HOST value: "elastic-es-http.logging.svc" - name: FLUENT_ELASTICSEARCH_PORT value: "9200" - name: FLUENT_ELASTICSEARCH_SCHEME value: "https" - name: FLUENT_ELASTICSEARCH_USER value: "elastic" - name: FLUENT_ELASTICSEARCH_PASSWORD value: "*****"