data-prepper icon indicating copy to clipboard operation
data-prepper copied to clipboard

Published 20 hours ago • verified publisher icon opensearch-project

[BUG] fluent-bit docker can't connect to data-prepper http 404

Open moorthi07 opened this issue 2 years ago • 7 comments

Describe the bug VM1 192.168.12.12:2021: Docker: data-prepper running

VM2: Fluent-bit docker: /fluent-bit/bin/fluent-bit -i cpu -t cpu -o http -p Host=192.168.12.12 -p Port=2021 -m * -p URI /data-prepper/log/ingest -p Format json

  1. See error in fluent-bit log: [2022/10/17 09:52:13] [ warn] [engine] failed to flush chunk '1-1666000332.243777239.flb', retry in 9 seconds: task_id=5, input=cpu.0 > output=http.0 (out_id=0)

[2022/10/17 09:52:14] [error] [output:http:http.0] 192.168.12.12:2021, HTTP status=404

Status: 404

Description: Not Found

And, What is the URI in this example? https://github.com/opensearch-project/data-prepper/blob/main/docs/log_analytics.md

URI /log/ingest Format json

Data-prepper works with 'Host = localhost' , not working from remote. Telnet able to connect to ip:2021

moorthi07 avatar Oct 18 '22 02:10 moorthi07

@moorthi07 Sorry for the delay. Thanks for your interest using Data Prepper! Looking at your VM2 fluentbit command, I noticed

URI /data-prepper/log/ingest

which I suppose should be

URI /log/ingest

assuming you already have port mapping configured in VM1. Let me know if it works.

Regarding your question on URI, the log ingestion URI is hardcoded to be /log/ingest. So the full endpoint is http(s)://<ip>:<port>/log/ingest.

chenqi0805 avatar Oct 20 '22 19:10 chenqi0805

Yes. I did try both. I tried the example as it is , then tried adding /data-prepper/. Both fails.

mapped -p 2021:2021 , hope that is the right port?

moorthi07 avatar Oct 20 '22 21:10 moorthi07

it fails even bypassing data-prepper.. with an error 'not an ssl data' . there is no way to specify https in fluent-bit config, tls on doesn't help either (with opensearch plugin )

moorthi07 avatar Oct 20 '22 21:10 moorthi07

it fails even bypassing data-prepper.. with an error 'not an ssl data' . there is no way to specify https in fluent-bit config, tls on doesn't help either (with opensearch plugin )

Not quite following here. It might be a networking issue but I would like to doublecheck a few things with you on VM1:

  1. What is the command you run data-prepper container?
  2. Could you share the pipeline.yml?
  3. Could you also share data-prepper container logs? Want to check if data-prepper has received the request.

chenqi0805 avatar Oct 20 '22 21:10 chenqi0805

Case 1: /fluent-bit/bin/fluent-bit -i cpu -t cpu -o http -p Host=192.168.64.9 -p Port=9200 -p uri /log/ingest -m * -p Format json fbit Log: [2022/10/20 21:01:26] [error] [http_client] broken connection to 192.168.64.9:9200 ?

[2022/10/20 21:01:26] [error] [output:http:http.0] could not flush records to 192.168.64.9:9200 (http_do=-1)

[2022/10/20 21:01:26] [ warn] [engine] failed to flush chunk '1-1666299685.592007836.flb', retry in 9 seconds: task_id=0, input=cpu.0 > output=http.0 (out_id=0)

OS log: 2022-10-20T21:02:25.615497392Z Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 504f5354202f20485454502f312e310d0a486f73743a203139322e3136382e36342e393a393230300d0a436f6e74656e742d4c656e6774683a203138390d0a436f6e74656e742d547970653a206170706c69636174696f6e2f6d73677061636b0d0a557365722d4167656e743a20466c75656e742d4269740d0a0d0a92d7006351b75a25d1ead889a56370755f70cb0000000000000000a6757365725f70cb0000000000000000a873797374656d5f70cb0000000000000000aa637075302e705f637075cb0000000000000000ab637075302e705f75736572cb0000000000000000ad637075302e705f73797374656dcb0000000000000000aa637075312e705f637075cb0000000000000000ab637075312e705f75736572cb0000000000000000ad637075312e705f73797374656dcb0000000000000000

2022-10-20T21:02:25.615531392Z at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1215) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]

2022-10-20T21:02:25.615561092Z at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]

2022-10-20T21:02:25.615590892Z at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]

2022-10-20T21:02:25.615629892Z at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]

2022-10-20T21:02:25.615673292Z ... 16 more

Case 2: /fluent-bit/bin/fluent-bit -i cpu -t cpu -o http -p Host=192.168.64.9 -p Port=2021 -p uri /log/ingest -m * -p Format json

fluent-bit log: Description: Not Found

[2022/10/20 21:04:50] [ warn] [engine] chunk '1-1666299878.629200136.flb' cannot be retried: task_id=2, input=cpu.0 > output=http.0

[2022/10/20 21:04:50] [error] [output:http:http.0] 192.168.64.9:2021, HTTP status=404

Status: 404

Description: Not Found

[2022/10/20 21:04:50] [ warn] [engine] failed to flush chunk '1-1666299889.591936636.flb', retry in 9 seconds: task_id=7, input=cpu.0 > output=http.0 (out_id=0)

Data-pepper log: 2022-10-20T20:55:06,639 [main] INFO org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [log-pipeline] - Initiating pipeline execution

2022-10-20T20:55:07,589 [main] WARN org.opensearch.dataprepper.plugins.source.loghttp.HTTPSource - Creating http source without SSL/TLS. This is not secure.

2022-10-20T20:55:07,590 [main] WARN org.opensearch.dataprepper.plugins.source.loghttp.HTTPSource - In order to set up TLS for the http source, go here: https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/http-source#ssl

2022-10-20T20:55:08,814 [main] INFO org.opensearch.dataprepper.plugins.source.loghttp.HTTPSource - Started http source on port 2021...

2022-10-20T20:55:08,816 [main] INFO org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [log-pipeline] - Submitting request to initiate the pipeline processing

2022-10-20T20:55:08,863 [main] INFO org.opensearch.dataprepper.pipeline.server.DataPrepperServer - Data Prepper server running at :4900

2022-10-20T20:55:11,861 [log-pipeline-processor-worker-1-thread-1] INFO org.opensearch.dataprepper.pipeline.ProcessWorker - log-pipeline Worker: No records received from buffer

moorthi07 avatar Oct 20 '22 22:10 moorthi07

Image opensearchproject/data-prepper:latest@sha256:a3df852cf1c5c073b0325eceb740b2fe9f5bd4e9da27e73bf16b3691892a8b03
Port configuration 0.0.0.0:2021 2021/tcp :::2021 2021/tcp

Image opensearchproject/data-prepper:latest@sha256:a3df852cf1c5c073b0325eceb740b2fe9f5bd4e9da27e73bf16b3691892a8b03 Port configuration 0.0.0.0:2021 2021/tcp :::2021 2021/tcp

moorthi07 avatar Oct 20 '22 22:10 moorthi07 

/data-prepper/examples/log-ingestion$ cat log_pipeline.yaml 
log-pipeline:
  source:
    http:
      ssl: false
      port: 2021
  processor:
    - grok:
        match:
          log: [ "%{COMMONAPACHELOG}" ]
  sink:
    - opensearch:
        hosts: [ "https://opensearch:9200" ]
        insecure: true
        username: admin
        password: admin
        index: mar_logs

moorthi07 avatar Oct 20 '22 22:10 moorthi07