fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard

Published 20 hours ago • verified publisher icon fluent

trace: add the capability for tracing messages in fluent-bit

Open pwhelan opened this issue 3 years ago • 10 comments

This branch adds the capability to trace messages as they flow through fluent-bit. To enable it the FLB_TRACE option must be enabled (cmake -DFLB_TRACE=On ..).

This feature still needs some work on the user experience so I am submitting it now for comment.

Right now traces are emitted records to an output that can be configured for tracing a designated input. Each input to be traced must be enabled and disabled separately at runtime via the embedded http server API.

Here is an example run:

./bin/fluent-bit -H -i dummy -F record_modifier -m '*' -p record="foo bar" -o stdout -m '*' -f 1
Fluent Bit v1.9.4
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/06/15 20:53:51] [ info] [fluent bit] version=1.9.4, commit=f7d4809714, pid=48369
[2022/06/15 20:53:51] [ info] [storage] version=1.2.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2022/06/15 20:53:51] [ info] [cmetrics] version=0.3.1
[2022/06/15 20:53:51] [ info] [output:stdout:stdout.0] worker #0 started
[2022/06/15 20:53:51] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2022/06/15 20:53:51] [ info] [sp] stream processor started
^[[1;9C[0] dummy.0: [1655340831.574631987, {"message"=>"dummy", "foo"=>"bar"}]
¡ª[0] dummy.0: [1655340832.574138421, {"message"=>"dummy", "foo"=>"bar"}]
[0] dummy.0: [1655340833.570861440, {"message"=>"dummy", "foo"=>"bar"}]
[0] dummy.0: [1655340834.572689498, {"message"=>"dummy", "foo"=>"bar"}]
[0] dummy.0: [1655340835.571385683, {"message"=>"dummy", "foo"=>"bar"}]
[0] dummy.0: [1655340836.571899742, {"message"=>"dummy", "foo"=>"bar"}]
[0] dummy.0: [1655340837.572128760, {"message"=>"dummy", "foo"=>"bar"}]
[0] dummy.0: [1655340838.574453985, {"message"=>"dummy", "foo"=>"bar"}]
[0] dummy.0: [1655340839.571634461, {"message"=>"dummy", "foo"=>"bar"}]
[0] dummy.0: [1655340840.574945603, {"message"=>"dummy", "foo"=>"bar"}]
[0] dummy.0: [1655340841.573376954, {"message"=>"dummy", "foo"=>"bar"}]
[0] dummy.0: [1655340842.571725472, {"message"=>"dummy", "foo"=>"bar"}]
[0] dummy.0: [1655340843.573333156, {"message"=>"dummy", "foo"=>"bar"}]
[0] dummy.0: [1655340844.573316841, {"message"=>"dummy", "foo"=>"bar"}]
[0] dummy.0: [1655340845.571674442, {"message"=>"dummy", "foo"=>"bar"}]
[0] dummy.0: [1655340846.573329167, {"message"=>"dummy", "foo"=>"bar"}]
[2022/06/15 20:54:08] [ info] [output:stdout:stdout.1] worker #0 started
[0] dummy.0: [1655340847.573561351, {"message"=>"dummy", "foo"=>"bar"}]
[{"date":1655340848.571074,"type":1,"trace_id":"trace.0","input_instance":"dummy.0","content":"ktcAYqp/MCIHpPiBp21lc3NhZ2WlZHVtbXk="}]
[{"date":1655340848.57138,"type":2,"time":1655340848.571381,"trace_id":"trace.0","filter_instance":"record_modifier.0","record_version":0,"record":"ktcAYqp/MCIHpPiCp21lc3NhZ2WlZHVtbXmjZm9vo2Jhcg=="}]
[0] dummy.0: [1655340848.570926328, {"message"=>"dummy", "foo"=>"bar"}]
[0] dummy.0: [1655340849.571375012, {"message"=>"dummy", "foo"=>"bar"}]
[{"date":1655340849.571467,"type":1,"trace_id":"trace.1","input_instance":"dummy.0","content":"ktcAYqp/MSIOfaSBp21lc3NhZ2WlZHVtbXk="}]
[{"date":1655340849.571738,"type":2,"time":1655340849.571738,"trace_id":"trace.1","filter_instance":"record_modifier.0","record_version":0,"record":"ktcAYqp/MSIOfaSCp21lc3NhZ2WlZHVtbXmjZm9vo2Jhcg=="}]
[0] dummy.0: [1655340850.573459321, {"message"=>"dummy", "foo"=>"bar"}]
[{"date":1655340850.57357,"type":1,"trace_id":"trace.2","input_instance":"dummy.0","content":"ktcAYqp/MiIuS3mBp21lc3NhZ2WlZHVtbXk="}]
[{"date":1655340850.573781,"type":2,"time":1655340850.573781,"trace_id":"trace.2","filter_instance":"record_modifier.0","record_version":0,"record":"ktcAYqp/MiIuS3mCp21lc3NhZ2WlZHVtbXmjZm9vo2Jhcg=="}]
[0] dummy.0: [1655340851.571983380, {"message"=>"dummy", "foo"=>"bar"}]
[{"date":1655340851.57206,"type":1,"trace_id":"trace.3","input_instance":"dummy.0","content":"ktcAYqp/MyIXxhSBp21lc3NhZ2WlZHVtbXk="}]
[{"date":1655340851.572114,"type":2,"time":1655340851.572115,"trace_id":"trace.3","filter_instance":"record_modifier.0","record_version":0,"record":"ktcAYqp/MyIXxhSCp21lc3NhZ2WlZHVtbXmjZm9vo2Jhcg=="}]
^C[2022/06/15 20:54:12] [engine] caught signal (SIGINT)
[{"date":1655340852.573585,"type":1,"trace_id":"trace.4","input_instance":"dummy.0","content":"ktcAYqp/NCIvkFGBp21lc3NhZ2WlZHVtbXk="}]
[{"date":1655340852.573689,"type":2,"time":1655340852.57369,"trace_id":"trace.4","filter_instance":"record_modifier.0","record_version":0,"record":"ktcAYqp/NCIvkFGCp21lc3NhZ2WlZHVtbXmjZm9vo2Jhcg=="}]

The actual traces were enabled using httpie:

http -v 127.0.0.1:2020/api/v1/trace/dummy.0 output=stdout prefix=trace. enable:=true params:='{"format":"json"}'
POST /api/v1/trace HTTP/1.1
Accept: application/json, */*;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 106
Content-Type: application/json
Host: 127.0.0.1:2020
User-Agent: HTTPie/2.6.0

{
    "enable": true,
    "output": "stdout",
    "params": {
        "format": "json"
    },
    "prefix": "trace."
}


HTTP/1.1 200 OK
Date: Thu, 16 Jun 2022 00:54:07 GMT
Server: Monkey/1.7.0
Transfer-Encoding: chunked

{"status":"ok","enabled":true}

The API is capable of turning on and off tracing for each and every input and configuring the output and its parameters. For the moment filtering is explicitly disabled for traces. This is to avoid filters with the wildcard match (Match *), etc...

Here is an example of a single trace for the input:

[
    {
        "date": 1655340850.57357,
        "type": 1,
        "trace_id": "trace.2",
        "input_instance": "dummy.0",
        "content": "ktcAYqp/MiIuS3mBp21lc3NhZ2WlZHVtbXk="
    }
]

and when it got modified by the filter:

[
    {
        "date": 1655340850.573781,
        "type": 2,
        "time": 1655340850.573781,
        "trace_id": "trace.2",
        "filter_instance": "record_modifier.0",
        "record_version": 0,
        "record": "ktcAYqp/MiIuS3mCp21lc3NhZ2WlZHVtbXmjZm9vo2Jhcg=="
    }
]

To disable a currently running trace:

$ http -v DELETE 127.0.0.1:2020/api/v1/trace/dummy.0
DELETE /api/v1/trace/dummy.0 HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 0
Host: 127.0.0.1:2020
User-Agent: HTTPie/3.2.1



HTTP/1.1 201 Created
Date: Mon, 20 Jun 2022 22:14:04 GMT
Server: Monkey/1.7.0
Transfer-Encoding: chunked

{"status":"ok"}

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

  • [ ] Example configuration file for the change
  • [ ] Debug log output from testing the change
  • [ ] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • [ ] Attached local packaging test output showing all targets (including any new ones) build.

Documentation

  • [ ] Documentation required for this feature

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

pwhelan avatar Jun 16 '22 01:06 pwhelan

Notes from the meeting 16/06/2022

  • [x] Extend the tracing endpoint to allow a DELETE method for disabling a tracing.
  • [x] Extend the POST method body with the following fields.
  • Input -> Inputs []string
  • Remove the "enable" option (as DELETE will take its place)
  • [x] Extend custom Calyptia to send traces to a specific endpoint. The endpoint URL will be composed of a prefix /pipeline_traces/{pipeline-id}/{session-id}, pipeline-id will be in the plugin configuration and session-id will come a parametrised tracing field.

niedbalski avatar Jun 16 '22 15:06 niedbalski

Feedback captured from meetup on 2022-06-16 for immediate consideration:

  • A way to find or hookup tracing by filter name or instance.
  • Add support for time expiration and capture limits as well as sampling.
  • Check to see if we can bind to an input based on the alias.
  • Make it possible to disable it at configuration time.

A lot of other useful features were mentioned that are relevant with respect to integration for distributed tracing and open telemtry:

  • Allow setting the trace_id of fluent-bit traces from an external trace_id from a message.
  • Integrate with Open Telemetry.

pwhelan avatar Jun 17 '22 14:06 pwhelan

@pwhelan need to report an error/failure for invalid settings - currently it accepts invalid plugin names to trace with no error (but does nothing with them at least).

patrick-stephens avatar Jun 20 '22 15:06 patrick-stephens

@pwhelan need to report an error/failure for invalid settings - currently it accepts invalid plugin names to trace with no error (but does nothing with them at least).

I am now reporting errors for:

  • enabling tracing on unknown inputs
  • deleting unknown inputs
  • using an unknown output for tracing
  • using invalid or unknown parameters for tracing.

This also comes with an update to the API. The input name is now passed as part of the URL, ie:

curl --header 'Content-Type: application/json' --data '{"enable":true, "input": "dummy.0", "output": "stdout", "params": { "format": "json" }, "prefix": "trace."}' "$REMOTE_HOST":2020/api/v1/trace

changes to

curl --header 'Content-Type: application/json' --data '{"enable":true, "output": "stdout", "params": { "format": "json" }, "prefix": "trace."}' "$REMOTE_HOST":2020/api/v1/trace/dummy.0

The DELETE method can be used to disable tracing via the same endpoint.

There is now also a /api/v1/traces endpoint which receives a inputs parameter as an array of strings for enabling (via POST) or disabling (via DELETE) tracing for multiple inputs at once.

pwhelan avatar Jun 20 '22 22:06 pwhelan

There are some memory problems that occur during shutdown within the HTTP server code that needs looking

valgrind ./bin/fluent-bit -H -i dummy -o stdout -m '*'  -f 1
==71756== Memcheck, a memory error detector
==71756== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==71756== Using Valgrind-3.19.0 and LibVEX; rerun with -h for copyright info
==71756== Command: ./bin/fluent-bit -H -i dummy -o stdout -m * -f 1
==71756==
Fluent Bit v1.9.6
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/07/11 19:11:05] [ info] [fluent bit] version=1.9.6, commit=edaeeb66fa, pid=71756
[2022/07/11 19:11:05] [ info] [storage] version=1.2.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2022/07/11 19:11:05] [ info] [cmetrics] version=0.3.5
[2022/07/11 19:11:05] [ info] [output:stdout:stdout.0] worker #0 started
[2022/07/11 19:11:05] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2022/07/11 19:11:05] [ info] [sp] stream processor started
[0] dummy.0: [1657581066.109139725, {"message"=>"dummy"}]
[0] dummy.0: [1657581067.105464643, {"message"=>"dummy"}]
[2022/07/11 19:11:09] [ info] [output:stdout:stdout.1] worker #0 started
[0] dummy.0: [1657581068.096188307, {"message"=>"dummy"}]
[0] dummy.0: [1657581069.096153709, {"message"=>"dummy"}]
[0] trace: [1657581069.099459110, {"type"=>1, "trace_id"=>"trace.0", "input_instance"=>"dummy.0", "records"=>[{"timestamp"=>1657581069, "record"=>{"message"=>"dummy"}}], "start_time"=>1657581069, "end_time"=>1657581069}]
[0] trace: [1657581069.108024135, {"type"=>3, "trace_id"=>"trace.0", "input_instance"=>"dummy.0", "records"=>[{"timestamp"=>1657581069, "record"=>{"message"=>"dummy"}}], "start_time"=>1657581069, "end_time"=>1657581069}]
[0] trace: [1657581070.101363922, {"type"=>1, "trace_id"=>"trace.1", "input_instance"=>"dummy.0", "records"=>[{"timestamp"=>1657581070, "record"=>{"message"=>"dummy"}}], "start_time"=>1657581070, "end_time"=>1657581070}]
[0] trace: [1657581070.101693741, {"type"=>3, "trace_id"=>"trace.1", "input_instance"=>"dummy.0", "records"=>[{"timestamp"=>1657581070, "record"=>{"message"=>"dummy"}}], "start_time"=>1657581070, "end_time"=>1657581070}]
[0] dummy.0: [1657581070.101179431, {"message"=>"dummy"}]
[0] trace: [1657581071.097740793, {"type"=>1, "trace_id"=>"trace.2", "input_instance"=>"dummy.0", "records"=>[{"timestamp"=>1657581071, "record"=>{"message"=>"dummy"}}], "start_time"=>1657581071, "end_time"=>1657581071}]
[0] trace: [1657581071.098112252, {"type"=>3, "trace_id"=>"trace.2", "input_instance"=>"dummy.0", "records"=>[{"timestamp"=>1657581071, "record"=>{"message"=>"dummy"}}], "start_time"=>1657581071, "end_time"=>1657581071}]
[0] dummy.0: [1657581071.097529982, {"message"=>"dummy"}]
[0] trace: [1657581072.096440497, {"type"=>1, "trace_id"=>"trace.3", "input_instance"=>"dummy.0", "records"=>[{"timestamp"=>1657581072, "record"=>{"message"=>"dummy"}}], "start_time"=>1657581072, "end_time"=>1657581072}]
[0] dummy.0: [1657581072.096259402, {"message"=>"dummy"}]
[0] trace: [1657581072.097001257, {"type"=>3, "trace_id"=>"trace.3", "input_instance"=>"dummy.0", "records"=>[{"timestamp"=>1657581072, "record"=>{"message"=>"dummy"}}], "start_time"=>1657581072, "end_time"=>1657581072}]
[0] trace: [1657581073.096802204, {"type"=>1, "trace_id"=>"trace.4", "input_instance"=>"dummy.0", "records"=>[{"timestamp"=>1657581073, "record"=>{"message"=>"dummy"}}], "start_time"=>1657581073, "end_time"=>1657581073}]
[0] dummy.0: [1657581073.096622511, {"message"=>"dummy"}]
[0] trace: [1657581073.097175365, {"type"=>3, "trace_id"=>"trace.4", "input_instance"=>"dummy.0", "records"=>[{"timestamp"=>1657581073, "record"=>{"message"=>"dummy"}}], "start_time"=>1657581073, "end_time"=>1657581073}]
[0] trace: [1657581074.096411406, {"type"=>1, "trace_id"=>"trace.5", "input_instance"=>"dummy.0", "records"=>[{"timestamp"=>1657581074, "record"=>{"message"=>"dummy"}}], "start_time"=>1657581074, "end_time"=>1657581074}]
[0] dummy.0: [1657581074.096230772, {"message"=>"dummy"}]
[0] trace: [1657581074.096769959, {"type"=>3, "trace_id"=>"trace.5", "input_instance"=>"dummy.0", "records"=>[{"timestamp"=>1657581074, "record"=>{"message"=>"dummy"}}], "start_time"=>1657581074, "end_time"=>1657581074}]
^C[2022/07/11 19:11:15] [engine] caught signal (SIGINT)
[0] dummy.0: [1657581075.098344323, {"message"=>"dummy"}]
[2022/07/11 19:11:15] [ warn] [engine] service will shutdown in max 5 seconds
[0] trace: [1657581075.098505811, {"type"=>1, "trace_id"=>"trace.6", "input_instance"=>"dummy.0", "records"=>[{"timestamp"=>1657581075, "record"=>{"message"=>"dummy"}}], "start_time"=>1657581075, "end_time"=>1657581075}]
[0] trace: [1657581075.098803118, {"type"=>3, "trace_id"=>"trace.6", "input_instance"=>"dummy.0", "records"=>[{"timestamp"=>1657581075, "record"=>{"message"=>"dummy"}}], "start_time"=>1657581075, "end_time"=>1657581075}]
[2022/07/11 19:11:16] [ info] [engine] service has stopped (0 pending tasks)
[2022/07/11 19:11:16] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2022/07/11 19:11:16] [ info] [output:stdout:stdout.0] thread worker #0 stopped
[2022/07/11 19:11:16] [ info] [output:stdout:stdout.1] thread worker #0 stopping...
[2022/07/11 19:11:16] [ info] [output:stdout:stdout.1] thread worker #0 stopped
==71756== Thread 2 flb-pipeline:
==71756== Invalid read of size 4
==71756==    at 0x548283: mk_stop (mk_lib.c:254)
==71756==    by 0x1E15FB: flb_hs_destroy (flb_hs.c:125)
==71756==    by 0x1A4C7D: flb_engine_shutdown (flb_engine.c:932)
==71756==    by 0x1A4832: flb_engine_start (flb_engine.c:823)
==71756==    by 0x17F83A: flb_lib_worker (flb_lib.c:626)
==71756==    by 0x4EB154C: start_thread (pthread_create.c:442)
==71756==    by 0x4F36873: clone (clone.S:100)
==71756==  Address 0x536558c is 396 bytes inside a block of size 744 free'd
==71756==    at 0x484826F: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==71756==    by 0x54CE7D: mk_mem_free (mk_memory.h:102)
==71756==    by 0x54D1FB: mk_config_free_all (mk_config.c:106)
==71756==    by 0x55D76B: mk_exit_all (monkey.c:218)
==71756==    by 0x5480A1: mk_lib_worker (mk_lib.c:174)
==71756==    by 0x4EB154C: start_thread (pthread_create.c:442)
==71756==    by 0x4F36873: clone (clone.S:100)
==71756==  Block was alloc'd at
==71756==    at 0x484AA73: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==71756==    by 0x55CF8A: mk_mem_alloc_z (mk_memory.h:70)
==71756==    by 0x55D43A: mk_server_create (monkey.c:89)
==71756==    by 0x547DCB: mk_create (mk_lib.c:61)
==71756==    by 0x1E13FE: flb_hs_create (flb_hs.c:77)
==71756==    by 0x1A4295: flb_engine_start (flb_engine.c:723)
==71756==    by 0x17F83A: flb_lib_worker (flb_lib.c:626)
==71756==    by 0x4EB154C: start_thread (pthread_create.c:442)
==71756==    by 0x4F36873: clone (clone.S:100)
==71756==
==71756==
==71756== HEAP SUMMARY:
==71756==     in use at exit: 107,098 bytes in 3,676 blocks
==71756==   total heap usage: 6,693 allocs, 3,017 frees, 8,773,084 bytes allocated
==71756==
==71756== LEAK SUMMARY:
==71756==    definitely lost: 56 bytes in 1 blocks
==71756==    indirectly lost: 0 bytes in 0 blocks
==71756==      possibly lost: 0 bytes in 0 blocks
==71756==    still reachable: 107,042 bytes in 3,675 blocks
==71756==         suppressed: 0 bytes in 0 blocks
==71756== Rerun with --leak-check=full to see details of leaked memory
==71756==
==71756== For lists of detected and suppressed errors, rerun with: -s
==71756== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

pwhelan avatar Jul 11 '22 23:07 pwhelan

I have found problems with several tests:

  • flb-rt-filter_kubernetes
    • missing files:
      • annotations-exclude_default_text
      • annotations-exclude_multiple-1_container-1
        • stdout
        • stderr
      • annotations-exclude_multiple-2_container-1
        • stdout
        • stderr
      • annotations-exclude_multiple-3_container-1
        • stdout
        • stderr
      • annotations-exclude_multiple-4_container-1
        • stdout
        • stderr
  • flb-rt-filter_lua
    • still unable to work out why it is failing
  • filter_multiline
    • flb_test_multiline_partial_message_concat
      • we get the pattern eventually.. but cb_check_result counts the non valid ones and then also expects to see the actual pattern at the wrong time...
    • multiline_buffered_two_record
      • ..
  • flb-rt-out_file
    • format_invalid
      • flb_start no longer checks configuration no longer checks and returns error when configuration options are invalid.

I have solutions for most of these problems. I have tried resurrecting the data files and changes from master to no avail. Any input would be highly appreciated. I will double check to see if other PRs on top of master have similar issues.

pwhelan avatar Jul 12 '22 18:07 pwhelan

Upon attempting to free the input and output instances I occasionally run into this error when running under valgrind:

[2022/07/13 13:17:54] [error] destroy trace context
[2022/07/13 13:17:54] [error] output callback exit
[2022/07/13 13:17:54] [error] stop thread pool
[2022/07/13 13:17:54] [ info] [output:stdout:stdout.1] thread worker #0 stopping...
[2022/07/13 13:17:54] [ info] [output:stdout:stdout.1] thread worker #0 stopped
[2022/07/13 13:17:54] [error] output instance destroy
==39973== Thread 2 flb-pipeline:
==39973== Invalid read of size 8
==39973==    at 0x1A6438: mk_list_del (mk_list.h:93)
==39973==    by 0x1A6777: flb_bucket_queue_delete_min (flb_bucket_queue.h:113)
==39973==    by 0x1A67C4: flb_bucket_queue_pop_min (flb_bucket_queue.h:122)
==39973==    by 0x1A8D48: flb_engine_start (flb_engine.c:793)
==39973==    by 0x18049A: flb_lib_worker (flb_lib.c:626)
==39973==    by 0x4EB154C: start_thread (pthread_create.c:442)
==39973==    by 0x4F36873: clone (clone.S:100)
==39973==  Address 0x9aa3bb8 is 56 bytes inside a block of size 688 free'd
==39973==    at 0x484826F: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x193504: flb_free (flb_mem.h:120)
==39973==    by 0x194680: flb_output_instance_destroy (flb_output.c:289)
==39973==    by 0x1CF409: flb_trace_chunk_context_destroy (flb_trace_chunk.c:214)
==39973==    by 0x1CECFA: flb_trace_chunk_destroy (flb_trace_chunk.c:46)
==39973==    by 0x1D9BCC: flb_input_chunk_destroy (flb_input_chunk.c:1108)
==39973==    by 0x1AAED4: flb_task_destroy (flb_task.c:471)
==39973==    by 0x1A6CD0: flb_task_users_release (flb_task.h:128)
==39973==    by 0x1A6D05: flb_task_users_dec (flb_task.h:146)
==39973==    by 0x1A74F3: handle_output_event (flb_engine.c:267)
==39973==    by 0x1A8C50: flb_engine_start (flb_engine.c:910)
==39973==    by 0x18049A: flb_lib_worker (flb_lib.c:626)
==39973==  Block was alloc'd at
==39973==    at 0x484AA73: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x1934EA: flb_calloc (flb_mem.h:88)
==39973==    by 0x194BC9: flb_output_new (flb_output.c:436)
==39973==    by 0x1CEF9A: flb_trace_chunk_context_new (flb_trace_chunk.c:111)
==39973==    by 0x2FE1F3: enable_trace_input (trace.c:65)
==39973==    by 0x2FE879: http_enable_trace (trace.c:232)
==39973==    by 0x2FF39A: cb_trace (trace.c:414)
==39973==    by 0x55D808: thread_cb_init_vars (mk_http_thread.c:112)
==39973==    by 0x56A526: co_init (amd64.c:117)
==39973== 
==39973== Invalid read of size 8
==39973==    at 0x1A6440: mk_list_del (mk_list.h:93)
==39973==    by 0x1A6777: flb_bucket_queue_delete_min (flb_bucket_queue.h:113)
==39973==    by 0x1A67C4: flb_bucket_queue_pop_min (flb_bucket_queue.h:122)
==39973==    by 0x1A8D48: flb_engine_start (flb_engine.c:793)
==39973==    by 0x18049A: flb_lib_worker (flb_lib.c:626)
==39973==    by 0x4EB154C: start_thread (pthread_create.c:442)
==39973==    by 0x4F36873: clone (clone.S:100)
==39973==  Address 0x9aa3bb0 is 48 bytes inside a block of size 688 free'd
==39973==    at 0x484826F: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x193504: flb_free (flb_mem.h:120)
==39973==    by 0x194680: flb_output_instance_destroy (flb_output.c:289)
==39973==    by 0x1CF409: flb_trace_chunk_context_destroy (flb_trace_chunk.c:214)
==39973==    by 0x1CECFA: flb_trace_chunk_destroy (flb_trace_chunk.c:46)
==39973==    by 0x1D9BCC: flb_input_chunk_destroy (flb_input_chunk.c:1108)
==39973==    by 0x1AAED4: flb_task_destroy (flb_task.c:471)
==39973==    by 0x1A6CD0: flb_task_users_release (flb_task.h:128)
==39973==    by 0x1A6D05: flb_task_users_dec (flb_task.h:146)
==39973==    by 0x1A74F3: handle_output_event (flb_engine.c:267)
==39973==    by 0x1A8C50: flb_engine_start (flb_engine.c:910)
==39973==    by 0x18049A: flb_lib_worker (flb_lib.c:626)
==39973==  Block was alloc'd at
==39973==    at 0x484AA73: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x1934EA: flb_calloc (flb_mem.h:88)
==39973==    by 0x194BC9: flb_output_new (flb_output.c:436)
==39973==    by 0x1CEF9A: flb_trace_chunk_context_new (flb_trace_chunk.c:111)
==39973==    by 0x2FE1F3: enable_trace_input (trace.c:65)
==39973==    by 0x2FE879: http_enable_trace (trace.c:232)
==39973==    by 0x2FF39A: cb_trace (trace.c:414)
==39973==    by 0x55D808: thread_cb_init_vars (mk_http_thread.c:112)
==39973==    by 0x56A526: co_init (amd64.c:117)
==39973== 
==39973== Invalid write of size 8
==39973==    at 0x1A6452: mk_list_del (mk_list.h:94)
==39973==    by 0x1A6777: flb_bucket_queue_delete_min (flb_bucket_queue.h:113)
==39973==    by 0x1A67C4: flb_bucket_queue_pop_min (flb_bucket_queue.h:122)
==39973==    by 0x1A8D48: flb_engine_start (flb_engine.c:793)
==39973==    by 0x18049A: flb_lib_worker (flb_lib.c:626)
==39973==    by 0x4EB154C: start_thread (pthread_create.c:442)
==39973==    by 0x4F36873: clone (clone.S:100)
==39973==  Address 0x9aa3bb0 is 48 bytes inside a block of size 688 free'd
==39973==    at 0x484826F: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x193504: flb_free (flb_mem.h:120)
==39973==    by 0x194680: flb_output_instance_destroy (flb_output.c:289)
==39973==    by 0x1CF409: flb_trace_chunk_context_destroy (flb_trace_chunk.c:214)
==39973==    by 0x1CECFA: flb_trace_chunk_destroy (flb_trace_chunk.c:46)
==39973==    by 0x1D9BCC: flb_input_chunk_destroy (flb_input_chunk.c:1108)
==39973==    by 0x1AAED4: flb_task_destroy (flb_task.c:471)
==39973==    by 0x1A6CD0: flb_task_users_release (flb_task.h:128)
==39973==    by 0x1A6D05: flb_task_users_dec (flb_task.h:146)
==39973==    by 0x1A74F3: handle_output_event (flb_engine.c:267)
==39973==    by 0x1A8C50: flb_engine_start (flb_engine.c:910)
==39973==    by 0x18049A: flb_lib_worker (flb_lib.c:626)
==39973==  Block was alloc'd at
==39973==    at 0x484AA73: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x1934EA: flb_calloc (flb_mem.h:88)
==39973==    by 0x194BC9: flb_output_new (flb_output.c:436)
==39973==    by 0x1CEF9A: flb_trace_chunk_context_new (flb_trace_chunk.c:111)
==39973==    by 0x2FE1F3: enable_trace_input (trace.c:65)
==39973==    by 0x2FE879: http_enable_trace (trace.c:232)
==39973==    by 0x2FF39A: cb_trace (trace.c:414)
==39973==    by 0x55D808: thread_cb_init_vars (mk_http_thread.c:112)
==39973==    by 0x56A526: co_init (amd64.c:117)
==39973== 
==39973== Invalid write of size 8
==39973==    at 0x1A645D: mk_list_del (mk_list.h:95)
==39973==    by 0x1A6777: flb_bucket_queue_delete_min (flb_bucket_queue.h:113)
==39973==    by 0x1A67C4: flb_bucket_queue_pop_min (flb_bucket_queue.h:122)
==39973==    by 0x1A8D48: flb_engine_start (flb_engine.c:793)
==39973==    by 0x18049A: flb_lib_worker (flb_lib.c:626)
==39973==    by 0x4EB154C: start_thread (pthread_create.c:442)
==39973==    by 0x4F36873: clone (clone.S:100)
==39973==  Address 0x9aa3bb8 is 56 bytes inside a block of size 688 free'd
==39973==    at 0x484826F: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x193504: flb_free (flb_mem.h:120)
==39973==    by 0x194680: flb_output_instance_destroy (flb_output.c:289)
==39973==    by 0x1CF409: flb_trace_chunk_context_destroy (flb_trace_chunk.c:214)
==39973==    by 0x1CECFA: flb_trace_chunk_destroy (flb_trace_chunk.c:46)
==39973==    by 0x1D9BCC: flb_input_chunk_destroy (flb_input_chunk.c:1108)
==39973==    by 0x1AAED4: flb_task_destroy (flb_task.c:471)
==39973==    by 0x1A6CD0: flb_task_users_release (flb_task.h:128)
==39973==    by 0x1A6D05: flb_task_users_dec (flb_task.h:146)
==39973==    by 0x1A74F3: handle_output_event (flb_engine.c:267)
==39973==    by 0x1A8C50: flb_engine_start (flb_engine.c:910)
==39973==    by 0x18049A: flb_lib_worker (flb_lib.c:626)
==39973==  Block was alloc'd at
==39973==    at 0x484AA73: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x1934EA: flb_calloc (flb_mem.h:88)
==39973==    by 0x194BC9: flb_output_new (flb_output.c:436)
==39973==    by 0x1CEF9A: flb_trace_chunk_context_new (flb_trace_chunk.c:111)
==39973==    by 0x2FE1F3: enable_trace_input (trace.c:65)
==39973==    by 0x2FE879: http_enable_trace (trace.c:232)
==39973==    by 0x2FF39A: cb_trace (trace.c:414)
==39973==    by 0x55D808: thread_cb_init_vars (mk_http_thread.c:112)
==39973==    by 0x56A526: co_init (amd64.c:117)
==39973== 
==39973== Invalid read of size 4
==39973==    at 0x1A8674: flb_engine_start (flb_engine.c:794)
==39973==    by 0x18049A: flb_lib_worker (flb_lib.c:626)
==39973==    by 0x4EB154C: start_thread (pthread_create.c:442)
==39973==    by 0x4F36873: clone (clone.S:100)
==39973==  Address 0x9aa3b84 is 4 bytes inside a block of size 688 free'd
==39973==    at 0x484826F: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x193504: flb_free (flb_mem.h:120)
==39973==    by 0x194680: flb_output_instance_destroy (flb_output.c:289)
==39973==    by 0x1CF409: flb_trace_chunk_context_destroy (flb_trace_chunk.c:214)
==39973==    by 0x1CECFA: flb_trace_chunk_destroy (flb_trace_chunk.c:46)
==39973==    by 0x1D9BCC: flb_input_chunk_destroy (flb_input_chunk.c:1108)
==39973==    by 0x1AAED4: flb_task_destroy (flb_task.c:471)
==39973==    by 0x1A6CD0: flb_task_users_release (flb_task.h:128)
==39973==    by 0x1A6D05: flb_task_users_dec (flb_task.h:146)
==39973==    by 0x1A74F3: handle_output_event (flb_engine.c:267)
==39973==    by 0x1A8C50: flb_engine_start (flb_engine.c:910)
==39973==    by 0x18049A: flb_lib_worker (flb_lib.c:626)
==39973==  Block was alloc'd at
==39973==    at 0x484AA73: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x1934EA: flb_calloc (flb_mem.h:88)
==39973==    by 0x194BC9: flb_output_new (flb_output.c:436)
==39973==    by 0x1CEF9A: flb_trace_chunk_context_new (flb_trace_chunk.c:111)
==39973==    by 0x2FE1F3: enable_trace_input (trace.c:65)
==39973==    by 0x2FE879: http_enable_trace (trace.c:232)
==39973==    by 0x2FF39A: cb_trace (trace.c:414)
==39973==    by 0x55D808: thread_cb_init_vars (mk_http_thread.c:112)
==39973==    by 0x56A526: co_init (amd64.c:117)
==39973== 
==39973== Invalid read of size 4
==39973==    at 0x1A8A57: flb_engine_start (flb_engine.c:868)
==39973==    by 0x18049A: flb_lib_worker (flb_lib.c:626)
==39973==    by 0x4EB154C: start_thread (pthread_create.c:442)
==39973==    by 0x4F36873: clone (clone.S:100)
==39973==  Address 0x9aa3b84 is 4 bytes inside a block of size 688 free'd
==39973==    at 0x484826F: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x193504: flb_free (flb_mem.h:120)
==39973==    by 0x194680: flb_output_instance_destroy (flb_output.c:289)
==39973==    by 0x1CF409: flb_trace_chunk_context_destroy (flb_trace_chunk.c:214)
==39973==    by 0x1CECFA: flb_trace_chunk_destroy (flb_trace_chunk.c:46)
==39973==    by 0x1D9BCC: flb_input_chunk_destroy (flb_input_chunk.c:1108)
==39973==    by 0x1AAED4: flb_task_destroy (flb_task.c:471)
==39973==    by 0x1A6CD0: flb_task_users_release (flb_task.h:128)
==39973==    by 0x1A6D05: flb_task_users_dec (flb_task.h:146)
==39973==    by 0x1A74F3: handle_output_event (flb_engine.c:267)
==39973==    by 0x1A8C50: flb_engine_start (flb_engine.c:910)
==39973==    by 0x18049A: flb_lib_worker (flb_lib.c:626)
==39973==  Block was alloc'd at
==39973==    at 0x484AA73: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x1934EA: flb_calloc (flb_mem.h:88)
==39973==    by 0x194BC9: flb_output_new (flb_output.c:436)
==39973==    by 0x1CEF9A: flb_trace_chunk_context_new (flb_trace_chunk.c:111)
==39973==    by 0x2FE1F3: enable_trace_input (trace.c:65)
==39973==    by 0x2FE879: http_enable_trace (trace.c:232)
==39973==    by 0x2FF39A: cb_trace (trace.c:414)
==39973==    by 0x55D808: thread_cb_init_vars (mk_http_thread.c:112)
==39973==    by 0x56A526: co_init (amd64.c:117)
==39973== 
==39973== Invalid read of size 4
==39973==    at 0x1A8A88: flb_engine_start (flb_engine.c:872)
==39973==    by 0x18049A: flb_lib_worker (flb_lib.c:626)
==39973==    by 0x4EB154C: start_thread (pthread_create.c:442)
==39973==    by 0x4F36873: clone (clone.S:100)
==39973==  Address 0x9aa3b84 is 4 bytes inside a block of size 688 free'd
==39973==    at 0x484826F: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x193504: flb_free (flb_mem.h:120)
==39973==    by 0x194680: flb_output_instance_destroy (flb_output.c:289)
==39973==    by 0x1CF409: flb_trace_chunk_context_destroy (flb_trace_chunk.c:214)
==39973==    by 0x1CECFA: flb_trace_chunk_destroy (flb_trace_chunk.c:46)
==39973==    by 0x1D9BCC: flb_input_chunk_destroy (flb_input_chunk.c:1108)
==39973==    by 0x1AAED4: flb_task_destroy (flb_task.c:471)
==39973==    by 0x1A6CD0: flb_task_users_release (flb_task.h:128)
==39973==    by 0x1A6D05: flb_task_users_dec (flb_task.h:146)
==39973==    by 0x1A74F3: handle_output_event (flb_engine.c:267)
==39973==    by 0x1A8C50: flb_engine_start (flb_engine.c:910)
==39973==    by 0x18049A: flb_lib_worker (flb_lib.c:626)
==39973==  Block was alloc'd at
==39973==    at 0x484AA73: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x1934EA: flb_calloc (flb_mem.h:88)
==39973==    by 0x194BC9: flb_output_new (flb_output.c:436)
==39973==    by 0x1CEF9A: flb_trace_chunk_context_new (flb_trace_chunk.c:111)
==39973==    by 0x2FE1F3: enable_trace_input (trace.c:65)
==39973==    by 0x2FE879: http_enable_trace (trace.c:232)
==39973==    by 0x2FF39A: cb_trace (trace.c:414)
==39973==    by 0x55D808: thread_cb_init_vars (mk_http_thread.c:112)
==39973==    by 0x56A526: co_init (amd64.c:117)
==39973== 
==39973== Invalid read of size 4
==39973==    at 0x1A8B3B: flb_engine_start (flb_engine.c:885)
==39973==    by 0x18049A: flb_lib_worker (flb_lib.c:626)
==39973==    by 0x4EB154C: start_thread (pthread_create.c:442)
==39973==    by 0x4F36873: clone (clone.S:100)
==39973==  Address 0x9aa3b84 is 4 bytes inside a block of size 688 free'd
==39973==    at 0x484826F: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x193504: flb_free (flb_mem.h:120)
==39973==    by 0x194680: flb_output_instance_destroy (flb_output.c:289)
==39973==    by 0x1CF409: flb_trace_chunk_context_destroy (flb_trace_chunk.c:214)
==39973==    by 0x1CECFA: flb_trace_chunk_destroy (flb_trace_chunk.c:46)
==39973==    by 0x1D9BCC: flb_input_chunk_destroy (flb_input_chunk.c:1108)
==39973==    by 0x1AAED4: flb_task_destroy (flb_task.c:471)
==39973==    by 0x1A6CD0: flb_task_users_release (flb_task.h:128)
==39973==    by 0x1A6D05: flb_task_users_dec (flb_task.h:146)
==39973==    by 0x1A74F3: handle_output_event (flb_engine.c:267)
==39973==    by 0x1A8C50: flb_engine_start (flb_engine.c:910)
==39973==    by 0x18049A: flb_lib_worker (flb_lib.c:626)
==39973==  Block was alloc'd at
==39973==    at 0x484AA73: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x1934EA: flb_calloc (flb_mem.h:88)
==39973==    by 0x194BC9: flb_output_new (flb_output.c:436)
==39973==    by 0x1CEF9A: flb_trace_chunk_context_new (flb_trace_chunk.c:111)
==39973==    by 0x2FE1F3: enable_trace_input (trace.c:65)
==39973==    by 0x2FE879: http_enable_trace (trace.c:232)
==39973==    by 0x2FF39A: cb_trace (trace.c:414)
==39973==    by 0x55D808: thread_cb_init_vars (mk_http_thread.c:112)
==39973==    by 0x56A526: co_init (amd64.c:117)
==39973== 
==39973== Invalid read of size 4
==39973==    at 0x1A8B66: flb_engine_start (flb_engine.c:888)
==39973==    by 0x18049A: flb_lib_worker (flb_lib.c:626)
==39973==    by 0x4EB154C: start_thread (pthread_create.c:442)
==39973==    by 0x4F36873: clone (clone.S:100)
==39973==  Address 0x9aa3b84 is 4 bytes inside a block of size 688 free'd
==39973==    at 0x484826F: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x193504: flb_free (flb_mem.h:120)
==39973==    by 0x194680: flb_output_instance_destroy (flb_output.c:289)
==39973==    by 0x1CF409: flb_trace_chunk_context_destroy (flb_trace_chunk.c:214)
==39973==    by 0x1CECFA: flb_trace_chunk_destroy (flb_trace_chunk.c:46)
==39973==    by 0x1D9BCC: flb_input_chunk_destroy (flb_input_chunk.c:1108)
==39973==    by 0x1AAED4: flb_task_destroy (flb_task.c:471)
==39973==    by 0x1A6CD0: flb_task_users_release (flb_task.h:128)
==39973==    by 0x1A6D05: flb_task_users_dec (flb_task.h:146)
==39973==    by 0x1A74F3: handle_output_event (flb_engine.c:267)
==39973==    by 0x1A8C50: flb_engine_start (flb_engine.c:910)
==39973==    by 0x18049A: flb_lib_worker (flb_lib.c:626)
==39973==  Block was alloc'd at
==39973==    at 0x484AA73: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x1934EA: flb_calloc (flb_mem.h:88)
==39973==    by 0x194BC9: flb_output_new (flb_output.c:436)
==39973==    by 0x1CEF9A: flb_trace_chunk_context_new (flb_trace_chunk.c:111)
==39973==    by 0x2FE1F3: enable_trace_input (trace.c:65)
==39973==    by 0x2FE879: http_enable_trace (trace.c:232)
==39973==    by 0x2FF39A: cb_trace (trace.c:414)
==39973==    by 0x55D808: thread_cb_init_vars (mk_http_thread.c:112)
==39973==    by 0x56A526: co_init (amd64.c:117)
==39973== 
==39973== Invalid read of size 4
==39973==    at 0x1A8C15: flb_engine_start (flb_engine.c:903)
==39973==    by 0x18049A: flb_lib_worker (flb_lib.c:626)
==39973==    by 0x4EB154C: start_thread (pthread_create.c:442)
==39973==    by 0x4F36873: clone (clone.S:100)
==39973==  Address 0x9aa3b84 is 4 bytes inside a block of size 688 free'd
==39973==    at 0x484826F: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x193504: flb_free (flb_mem.h:120)
==39973==    by 0x194680: flb_output_instance_destroy (flb_output.c:289)
==39973==    by 0x1CF409: flb_trace_chunk_context_destroy (flb_trace_chunk.c:214)
==39973==    by 0x1CECFA: flb_trace_chunk_destroy (flb_trace_chunk.c:46)
==39973==    by 0x1D9BCC: flb_input_chunk_destroy (flb_input_chunk.c:1108)
==39973==    by 0x1AAED4: flb_task_destroy (flb_task.c:471)
==39973==    by 0x1A6CD0: flb_task_users_release (flb_task.h:128)
==39973==    by 0x1A6D05: flb_task_users_dec (flb_task.h:146)
==39973==    by 0x1A74F3: handle_output_event (flb_engine.c:267)
==39973==    by 0x1A8C50: flb_engine_start (flb_engine.c:910)
==39973==    by 0x18049A: flb_lib_worker (flb_lib.c:626)
==39973==  Block was alloc'd at
==39973==    at 0x484AA73: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x1934EA: flb_calloc (flb_mem.h:88)
==39973==    by 0x194BC9: flb_output_new (flb_output.c:436)
==39973==    by 0x1CEF9A: flb_trace_chunk_context_new (flb_trace_chunk.c:111)
==39973==    by 0x2FE1F3: enable_trace_input (trace.c:65)
==39973==    by 0x2FE879: http_enable_trace (trace.c:232)
==39973==    by 0x2FF39A: cb_trace (trace.c:414)
==39973==    by 0x55D808: thread_cb_init_vars (mk_http_thread.c:112)
==39973==    by 0x56A526: co_init (amd64.c:117)
==39973== 
==39973== Invalid read of size 4
==39973==    at 0x1A8C37: flb_engine_start (flb_engine.c:910)
==39973==    by 0x18049A: flb_lib_worker (flb_lib.c:626)
==39973==    by 0x4EB154C: start_thread (pthread_create.c:442)
==39973==    by 0x4F36873: clone (clone.S:100)
==39973==  Address 0x9aa3b80 is 0 bytes inside a block of size 688 free'd
==39973==    at 0x484826F: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x193504: flb_free (flb_mem.h:120)
==39973==    by 0x194680: flb_output_instance_destroy (flb_output.c:289)
==39973==    by 0x1CF409: flb_trace_chunk_context_destroy (flb_trace_chunk.c:214)
==39973==    by 0x1CECFA: flb_trace_chunk_destroy (flb_trace_chunk.c:46)
==39973==    by 0x1D9BCC: flb_input_chunk_destroy (flb_input_chunk.c:1108)
==39973==    by 0x1AAED4: flb_task_destroy (flb_task.c:471)
==39973==    by 0x1A6CD0: flb_task_users_release (flb_task.h:128)
==39973==    by 0x1A6D05: flb_task_users_dec (flb_task.h:146)
==39973==    by 0x1A74F3: handle_output_event (flb_engine.c:267)
==39973==    by 0x1A8C50: flb_engine_start (flb_engine.c:910)
==39973==    by 0x18049A: flb_lib_worker (flb_lib.c:626)
==39973==  Block was alloc'd at
==39973==    at 0x484AA73: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x1934EA: flb_calloc (flb_mem.h:88)
==39973==    by 0x194BC9: flb_output_new (flb_output.c:436)
==39973==    by 0x1CEF9A: flb_trace_chunk_context_new (flb_trace_chunk.c:111)
==39973==    by 0x2FE1F3: enable_trace_input (trace.c:65)
==39973==    by 0x2FE879: http_enable_trace (trace.c:232)
==39973==    by 0x2FF39A: cb_trace (trace.c:414)
==39973==    by 0x55D808: thread_cb_init_vars (mk_http_thread.c:112)
==39973==    by 0x56A526: co_init (amd64.c:117)
==39973==

I suspect there is a race condition somewhere since I am effectively destroying the output instance from some input thread.

pwhelan avatar Jul 13 '22 17:07 pwhelan

why the code is trying to destroy an output instance ?, is this output instance something you created ?

==39973==    at 0x484826F: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==39973==    by 0x193504: flb_free (flb_mem.h:120)
==39973==    by 0x194680: flb_output_instance_destroy (flb_output.c:289)
==39973==    by 0x1CF409: flb_trace_chunk_context_destroy (flb_trace_chunk.c:214)
==39973==    by 0x1CECFA: flb_trace_chunk_destroy (flb_trace_chunk.c:46)
==39973==    by 0x1D9BCC: flb_input_chunk_destroy (flb_input_chunk.c:1108)
==39973==    by 0x1AAED4: flb_task_destroy (flb_task.c:471)
==39973==    by 0x1A6CD0: flb_task_users_release (flb_task.h:128)
==39973==    by 0x1A6D05: flb_task_users_dec (flb_task.h:146)
==39973==    by 0x1A74F3: handle_output_event (flb_engine.c:267)

but I see the code is commented now...

https://github.com/fluent/fluent-bit/pull/5580/files#diff-402f3928ceadf00d9c999c40e5af65700cab4e5f5e3888a3bac780725d288a6aR191-R219

CI is failing in different parts..

edsiper avatar Jul 16 '22 07:07 edsiper

why the code is trying to destroy an output instance ?, is this output instance something you created ? but I see the code is commented now...

I have pending changes which should fix all the problems with allocation. That ouput and input instance are the ones used by tracing to push its records. The new code creates a completely isolated and separate fluent-bit pipeline that is created and destroyed when trace is enabled and disabled.

pwhelan avatar Jul 16 '22 20:07 pwhelan

@pwhelan ack. review on hold until PR gets the latest changes.

edsiper avatar Jul 17 '22 23:07 edsiper