fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard

Published 20 hours ago • verified publisher icon fluent

plugins: New Azure Kusto Output Plugin

Open papigers opened this issue 2 years ago • 37 comments

Adds an output plugin for Azure Data Explorer (aka Kusto). #5591

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

  • [x] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • [N/A] Attached local packaging test output showing all targets (including any new ones) build.

Documentation

  • [x] Documentation required for this feature

Will create documentation PR in the upcoming days

Backporting

  • [N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

papigers avatar Jun 27 '22 00:06 papigers

Sample configuration (https://pastebin.com/B6wT4aPP)

[SERVICE]
  Daemon          off
  Flush 5

[INPUT]
  Name dummy
  Tag    dummy.log
  Dummy {"top": {".dotted": "value"}}

[OUTPUT]
  Match *
  Name azure_kusto
  Tenant_Id <azure_ad_directory/tenant_id>
  Client_Id <azure_ad_app_id>
  Client_Secret <secret>
  Ingestion_Endpoint https://ingest-clustername.northeurope.kusto.windows.net
  Database_Name MyDatabase
  Table_Name MyTable
  Include_Tag_Key On
  Tag_Key tag
  Include_Time_Key On
  Time_Key time

Debug output (https://pastebin.com/SWm8tFnL)

fluent-bit -c ./fluent-bit.conf
Fluent Bit v1.9.6
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/06/27 02:37:18] [ info] [fluent bit] version=1.9.6, commit=, pid=26487
[2022/06/27 02:37:18] [ info] [storage] version=1.2.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2022/06/27 02:37:18] [ info] [cmetrics] version=0.3.4
[2022/06/27 02:37:18] [ info] [output:azure_kusto:azure_kusto.0] endpoint='https://ingest-kvcv10n0jt157umdjvadsq.northeurope.dev.kusto.windows.net', database='MyDatabase', table='MyTable'
[2022/06/27 02:37:18] [ info] [oauth2] HTTP Status=200
[2022/06/27 02:37:18] [ info] [oauth2] access token from 'login.microsoftonline.com:443' retrieved
[2022/06/27 02:37:18] [ info] [output:azure_kusto:azure_kusto.0] loading kusto ingestion resourcs
[2022/06/27 02:37:18] [debug] [output:azure_kusto:azure_kusto.0] Kusto ingestion command request http_do=0, HTTP Status: 200
[2022/06/27 02:37:18] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: TempStorage 
[2022/06/27 02:37:18] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: SecuredReadyForAggregationQueue 
[2022/06/27 02:37:18] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: SecuredReadyForAggregationQueue 
[2022/06/27 02:37:18] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: SecuredReadyForAggregationQueue 
[2022/06/27 02:37:18] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: SecuredReadyForAggregationQueue 
[2022/06/27 02:37:18] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: SuccessfulIngestionsQueue 
[2022/06/27 02:37:18] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: SuccessfulIngestionsQueue 
[2022/06/27 02:37:18] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: SuccessfulIngestionsQueue 
[2022/06/27 02:37:18] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: SuccessfulIngestionsQueue 
[2022/06/27 02:37:18] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: FailedIngestionsQueue 
[2022/06/27 02:37:18] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: FailedIngestionsQueue 
[2022/06/27 02:37:18] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: FailedIngestionsQueue 
[2022/06/27 02:37:18] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: FailedIngestionsQueue 
[2022/06/27 02:37:18] [debug] [output:azure_kusto:azure_kusto.0] found resource of type: IngestionsStatusTable 
[2022/06/27 02:37:18] [debug] [output:azure_kusto:azure_kusto.0] parsed 1 blob resources and 4 queue resources
[2022/06/27 02:37:19] [debug] [output:azure_kusto:azure_kusto.0] Kusto ingestion command request http_do=0, HTTP Status: 200
[2022/06/27 02:37:19] [debug] [output:azure_kusto:azure_kusto.0] parsed kusto identity token: '<redacted_jwt_token>'
[2022/06/27 02:37:19] [ info] [sp] stream processor started
[2022/06/27 02:37:23] [debug] [output:azure_kusto:azure_kusto.0] resources are already loaded and are not stale
[2022/06/27 02:37:23] [debug] [output:azure_kusto:azure_kusto.0] uploading payload to blob uri: /kvcv10n0jt157umdjvadsq-20220626-ingestdata-e5c334ee145d4b4-0/flb__MyDatabase__MyTable__ZHVtbXkubG9nPyY9XjM__1656286643631.multijson?<redacted_blob_sas_token>
[2022/06/27 02:37:23] [debug] [output:azure_kusto:azure_kusto.0] kusto blob upload request http_do=0, HTTP Status: 201
[2022/06/27 02:37:23] [debug] [output:azure_kusto:azure_kusto.0] created blob https://n58virtualdms.blob.core.windows.net/kvcv10n0jt157umdjvadsq-20220626-ingestdata-e5c334ee145d4b4-0/flb__MyDatabase__MyTable__ZHVtbXkubG9nPyY9XjM__1656286643631.multijson?<redacted_blob_sas_token>
[2022/06/27 02:37:24] [debug] [output:azure_kusto:azure_kusto.0] created ingestion message:
{"Id": "34aca4f2-86a6-606b-2fa3-e1bd7e8ffb11", "BlobPath": "https://n58virtualdms.blob.core.windows.net/kvcv10n0jt157umdjvadsq-20220626-ingestdata-e5c334ee145d4b4-0/flb__MyDatabase__MyTable__ZHVtbXkubG9nPyY9XjM__1656286643631.multijson?<redacted_blob_sas_token>", "RawDataSize": 393, "DatabaseName": "MyDatabase", "TableName": "MyTable","AdditionalProperties": { "format": "multijson", "authorizationContext": "<redacted_jwt_token>", "jsonMappingReference": "" }}
[2022/06/27 02:37:24] [debug] [output:azure_kusto:azure_kusto.0] kusto queue request http_do=0, HTTP Status: 201
[2022/06/27 02:37:28] [debug] [output:azure_kusto:azure_kusto.0] resources are already loaded and are not stale
[2022/06/27 02:37:28] [debug] [output:azure_kusto:azure_kusto.0] uploading payload to blob uri: /kvcv10n0jt157umdjvadsq-20220626-ingestdata-e5c334ee145d4b4-0/flb__MyDatabase__MyTable__ZHVtbXkubG9nPyY9XjM__1656286648631.multijson?<redacted_blob_sas_token>
[2022/06/27 02:37:28] [debug] [output:azure_kusto:azure_kusto.0] kusto blob upload request http_do=0, HTTP Status: 201
[2022/06/27 02:37:28] [debug] [output:azure_kusto:azure_kusto.0] created blob https://n58virtualdms.blob.core.windows.net/kvcv10n0jt157umdjvadsq-20220626-ingestdata-e5c334ee145d4b4-0/flb__MyDatabase__MyTable__ZHVtbXkubG9nPyY9XjM__1656286648631.multijson?<redacted_blob_sas_token>
[2022/06/27 02:37:28] [debug] [output:azure_kusto:azure_kusto.0] created ingestion message:
{"Id": "03a0eac0-a4cb-0d07-1287-83110c091796", "BlobPath": "https://n58virtualdms.blob.core.windows.net/kvcv10n0jt157umdjvadsq-20220626-ingestdata-e5c334ee145d4b4-0/flb__MyDatabase__MyTable__ZHVtbXkubG9nPyY9XjM__1656286648631.multijson?<redacted_blob_sas_token>", "RawDataSize": 491, "DatabaseName": "MyDatabase", "TableName": "MyTable","AdditionalProperties": { "format": "multijson", "authorizationContext": "<redacted_jwt_token>, "jsonMappingReference": "" }}
[2022/06/27 02:37:28] [debug] [output:azure_kusto:azure_kusto.0] kusto queue request http_do=0, HTTP Status: 201
[2022/06/27 02:37:33] [debug] [output:azure_kusto:azure_kusto.0] resources are already loaded and are not stale
[2022/06/27 02:37:33] [debug] [output:azure_kusto:azure_kusto.0] uploading payload to blob uri: /kvcv10n0jt157umdjvadsq-20220626-ingestdata-e5c334ee145d4b4-0/flb__MyDatabase__MyTable__ZHVtbXkubG9nPyY9XjM__1656286653632.multijson<redacted_blob_sas_token>
[2022/06/27 02:37:33] [debug] [output:azure_kusto:azure_kusto.0] kusto blob upload request http_do=0, HTTP Status: 201
[2022/06/27 02:37:33] [debug] [output:azure_kusto:azure_kusto.0] created blob https://n58virtualdms.blob.core.windows.net/kvcv10n0jt157umdjvadsq-20220626-ingestdata-e5c334ee145d4b4-0/flb__MyDatabase__MyTable__ZHVtbXkubG9nPyY9XjM__1656286653632.multijson<redacted_blob_sas_token>
[2022/06/27 02:37:33] [debug] [output:azure_kusto:azure_kusto.0] created ingestion message:
{"Id": "39d53c40-5a4f-8046-fdfd-3a94be4d62eb", "BlobPath": "https://n58virtualdms.blob.core.windows.net/kvcv10n0jt157umdjvadsq-20220626-ingestdata-e5c334ee145d4b4-0/flb__MyDatabase__MyTable__ZHVtbXkubG9nPyY9XjM__1656286653632.multijson<redacted_blob_sas_token>", "RawDataSize": 491, "DatabaseName": "MyDatabase", "TableName": "MyTable","AdditionalProperties": { "format": "multijson", "authorizationContext": "<redacted_jwt_token>", "jsonMappingReference": "" }}
[2022/06/27 02:37:33] [debug] [output:azure_kusto:azure_kusto.0] kusto queue request http_do=0, HTTP Status: 201
[2022/06/27 02:37:38] [debug] [output:azure_kusto:azure_kusto.0] resources are already loaded and are not stale
[2022/06/27 02:37:38] [debug] [output:azure_kusto:azure_kusto.0] uploading payload to blob uri: /kvcv10n0jt157umdjvadsq-20220626-ingestdata-e5c334ee145d4b4-0/flb__MyDatabase__MyTable__ZHVtbXkubG9nPyY9XjM__1656286658632.multijson<redacted_blob_sas_token>
[2022/06/27 02:37:38] [debug] [output:azure_kusto:azure_kusto.0] kusto blob upload request http_do=0, HTTP Status: 201
[2022/06/27 02:37:38] [debug] [output:azure_kusto:azure_kusto.0] created blob https://n58virtualdms.blob.core.windows.net/kvcv10n0jt157umdjvadsq-20220626-ingestdata-e5c334ee145d4b4-0/flb__MyDatabase__MyTable__ZHVtbXkubG9nPyY9XjM__1656286658632.multijson<redacted_blob_sas_token>
[2022/06/27 02:37:38] [debug] [output:azure_kusto:azure_kusto.0] created ingestion message:
{"Id": "5d7f87bb-96d4-fed3-8d7f-2c96ce287ced", "BlobPath": "https://n58virtualdms.blob.core.windows.net/kvcv10n0jt157umdjvadsq-20220626-ingestdata-e5c334ee145d4b4-0/flb__MyDatabase__MyTable__ZHVtbXkubG9nPyY9XjM__1656286658632.multijson<redacted_blob_sas_token>", "RawDataSize": 491, "DatabaseName": "MyDatabase", "TableName": "MyTable","AdditionalProperties": { "format": "multijson", "authorizationContext": "<redacted_jwt_token>", "jsonMappingReference": "" }}
[2022/06/27 02:37:38] [debug] [output:azure_kusto:azure_kusto.0] kusto queue request http_do=0, HTTP Status: 201
^C[2022/06/27 02:37:41] [engine] caught signal (SIGINT)
[2022/06/27 02:37:41] [ warn] [engine] service will shutdown in max 5 seconds
[2022/06/27 02:37:41] [debug] [output:azure_kusto:azure_kusto.0] resources are already loaded and are not stale
[2022/06/27 02:37:41] [debug] [output:azure_kusto:azure_kusto.0] uploading payload to blob uri: /kvcv10n0jt157umdjvadsq-20220626-ingestdata-e5c334ee145d4b4-0/flb__MyDatabase__MyTable__ZHVtbXkubG9nPyY9XjM__1656286661962.multijson<redacted_blob_sas_token>
[2022/06/27 02:37:42] [debug] [output:azure_kusto:azure_kusto.0] kusto blob upload request http_do=0, HTTP Status: 201
[2022/06/27 02:37:42] [debug] [output:azure_kusto:azure_kusto.0] created blob https://n58virtualdms.blob.core.windows.net/kvcv10n0jt157umdjvadsq-20220626-ingestdata-e5c334ee145d4b4-0/flb__MyDatabase__MyTable__ZHVtbXkubG9nPyY9XjM__1656286661962.multijson<redacted_blob_sas_token>
[2022/06/27 02:37:42] [debug] [output:azure_kusto:azure_kusto.0] created ingestion message:
{"Id": "74adedd3-d33e-a461-c838-4825bfa7d07d", "BlobPath": "https://n58virtualdms.blob.core.windows.net/kvcv10n0jt157umdjvadsq-20220626-ingestdata-e5c334ee145d4b4-0/flb__MyDatabase__MyTable__ZHVtbXkubG9nPyY9XjM__1656286661962.multijson<redacted_blob_sas_token>", "RawDataSize": 393, "DatabaseName": "MyDatabase", "TableName": "MyTable","AdditionalProperties": { "format": "multijson", "authorizationContext": "<redacted_jwt_token>", "jsonMappingReference": "" }}
[2022/06/27 02:37:42] [debug] [output:azure_kusto:azure_kusto.0] kusto queue request http_do=0, HTTP Status: 201
[2022/06/27 02:37:42] [ info] [engine] service has stopped (0 pending tasks)

Valgrind output (https://pastebin.com/gtWJbRzv)

valgrind --tool=memcheck --leak-check=full fluent-bit -c ./fluent-bit.conf
==17725== Memcheck, a memory error detector
==17725== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==17725== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==17725== Command: fluent-bit -c ./fluent-bit.conf
==17725== 
Fluent Bit v1.9.6
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/06/27 03:31:26] [ info] [fluent bit] version=1.9.6, commit=, pid=17725
[2022/06/27 03:31:26] [ info] [storage] version=1.2.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2022/06/27 03:31:26] [ info] [cmetrics] version=0.3.4
[2022/06/27 03:31:26] [ info] [output:azure_kusto:azure_kusto.0] endpoint='https://ingest-kvcv10n0jt157umdjvadsq.northeurope.dev.kusto.windows.net', database='MyDatabase', table='MyTable'
[2022/06/27 03:31:28] [ info] [oauth2] HTTP Status=200
[2022/06/27 03:31:28] [ info] [oauth2] access token from 'login.microsoftonline.com:443' retrieved
[2022/06/27 03:31:28] [ info] [output:azure_kusto:azure_kusto.0] loading kusto ingestion resourcs
[2022/06/27 03:31:29] [ info] [sp] stream processor started
==17725== Warning: client switching stacks?  SP change: 0x5c9b808 --> 0x68fe5d0
==17725==          to suppress, use: --max-stackframe=12987848 or greater
==17725== Warning: client switching stacks?  SP change: 0x68fe528 --> 0x5c9b808
==17725==          to suppress, use: --max-stackframe=12987680 or greater
==17725== Warning: client switching stacks?  SP change: 0x5c9ba38 --> 0x68fe528
==17725==          to suppress, use: --max-stackframe=12987120 or greater
==17725==          further instances of this message will not be shown.
==17725== Thread 2 flb-pipeline:
==17725== Invalid read of size 1
==17725==    at 0x484ED24: strlen (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==17725==  Address 0x6b2a604 is 0 bytes after a block of size 36 alloc'd
==17725==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==17725== 
^C[2022/06/27 03:31:41] [engine] caught signal (SIGINT)
[2022/06/27 03:31:41] [ warn] [engine] service will shutdown in max 5 seconds
[2022/06/27 03:31:41] [ info] [task] dummy/dummy.0 has 1 pending task(s):
[2022/06/27 03:31:41] [ info] [task]   task_id=0 still running on route(s): azure_kusto/azure_kusto.0 
[2022/06/27 03:31:42] [ info] [engine] service has stopped (0 pending tasks)
==17725== 
==17725== HEAP SUMMARY:
==17725==     in use at exit: 762,568 bytes in 5,806 blocks
==17725==   total heap usage: 59,677 allocs, 53,871 frees, 9,488,410 bytes allocated
==17725== 
==17725== Thread 1:
==17725== 610 bytes in 3 blocks are definitely lost in loss record 733 of 819
==17725==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==17725== 
==17725== LEAK SUMMARY:
==17725==    definitely lost: 610 bytes in 3 blocks
==17725==    indirectly lost: 0 bytes in 0 blocks
==17725==      possibly lost: 0 bytes in 0 blocks
==17725==    still reachable: 761,958 bytes in 5,803 blocks
==17725==         suppressed: 0 bytes in 0 blocks
==17725== Reachable blocks (those to which a pointer was found) are not shown.
==17725== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==17725== 
==17725== For lists of detected and suppressed errors, rerun with: -s
==17725== ERROR SUMMARY: 7 errors from 2 contexts (suppressed: 0 from 0)

papigers avatar Jun 27 '22 00:06 papigers

@papigers can you raise a docs PR and link it here for the new plugin too? I can't comment on the actual changes here but I've approved the workflows to run - the docs PR is needed so we can include in a release after merging this.

@agup006 looks like a good addition for 2.0 maybe?

patrick-stephens avatar Jun 27 '22 10:06 patrick-stephens

@papigers it's great to have the new plugin but probably need some help on integration testing as well over at fluent/fluent-bit-ci. Can you link a PR from there with the integration tests as well? Ideally we can test in isolation and not requiring more infrastructure set up but feel free to ping me or @niedbalski to discuss.

patrick-stephens avatar Jun 27 '22 10:06 patrick-stephens

@patrick-stephens

can you raise a docs PR and link it here for the new plugin too? I can't comment on the actual changes here but I've approved the workflows to run - the docs PR is needed so we can include in a release after merging this.

Hopefully will find a time for it today/tomorrow.

it's great to have the new plugin but probably need some help on integration testing as well over at fluent/fluent-bit-ci. Can you link a PR from there with the integration tests as well? Ideally we can test in isolation and not requiring more infrastructure set up but feel free to ping me or @niedbalski to discuss.

Unfortunately, kusto doesn't have an emulator to run against locally, but we could probably utilize their free tier offering. We would also need to create an azure account (no subscription/payment needed AFAIK), and then create an oauth application in that directory, and use its credentials in the plugin's configuration.

papigers avatar Jun 27 '22 12:06 papigers

Unfortunately, kusto doesn't have an emulator to run against locally, but we could probably utilize their free tier offering. We would also need to create an azure account (no subscription/payment needed AFAIK), and then create an oauth application in that directory, and use its credentials in the plugin's configuration.

Would it be possible to pull together a test then, assuming we have access, so we can provide it? We can sort out the access separately then and just have your test case hook into it.

patrick-stephens avatar Jun 27 '22 12:06 patrick-stephens

Sure thing, I'll update as soon as I have something ready.

papigers avatar Jun 27 '22 13:06 papigers

Thanks @papigers , feel free to open a PR on the fluent ci repo.

An example one with a linked integration test: https://github.com/fluent/fluent-bit/pull/5557

patrick-stephens avatar Jun 27 '22 13:06 patrick-stephens

Hey @patrick-stephens, sorry for the delay, I've created this CI PR: https://github.com/fluent/fluent-bit-ci/pull/66

Please tell me what you think. I'm still planning to work on the documentation as well, hopefully I'll find the time over the weekend.

papigers avatar Jul 06 '22 00:07 papigers

Super awesome is what I think @papigers , really appreciate it! Once we land the CI PR we can run it with this change.

We may have to look at how we handle running it for PRs without your plugin though. Don't want failures for previous PRs.

patrick-stephens avatar Jul 06 '22 05:07 patrick-stephens

@papigers this is awesome, want to present this at our community meeting this Thursday? https://docs.google.com/document/d/1vJvsn8E0SanLO1R0X3RC1qTw0XQK_7q75sZ8IbWAu-g/edit#

agup006 avatar Jul 06 '22 06:07 agup006

@agup006 wow, thanks a lot for the offer. This Thursday I won't be available for that, but maybe the meeting after that? If that works for you I'd also appreciate a breif/demo of how it usually goes so I would come prepared :)

papigers avatar Jul 06 '22 20:07 papigers

@agup006 wow, thanks a lot for the offer. This Thursday I won't be available for that, but maybe the meeting after that? If that works for you I'd also appreciate a breif/demo of how it usually goes so I would come prepared :)

I think that's fine for the one after if we can schedule you in - the dates should be on the meet up page and I'll add you to the agenda. As to how it goes, it's pretty casual and there are a few recorded ones in the linked agenda doc if you want to watch a previous one. We should really get them up on Youtube or similar @agup006 .

patrick-stephens avatar Jul 07 '22 11:07 patrick-stephens

Could you please explain the logic behind the flb_upstream_node_destroy correction?

@leonardo-albertovich Discovered by running valgrind.

==53772== Invalid read of size 8
==53772==    at 0x1BEF2A: flb_tls_session_destroy (flb_tls.c:439)
==53772==    by 0x1A6E1A: destroy_conn (flb_upstream.c:487)
==53772==    by 0x1A7287: flb_upstream_destroy (flb_upstream.c:605)
==53772==    by 0x2D6CE5: flb_upstream_node_destroy (flb_upstream_node.c:193)
==53772==    by 0x2D5892: flb_upstream_ha_destroy (flb_upstream_ha.c:72)
==53772==    by 0x22F83B: flb_azure_kusto_resources_clear (azure_kusto_conf.c:125)
==53772==    by 0x230B7F: flb_azure_kusto_resources_destroy (azure_kusto_conf.c:516)
==53772==    by 0x231263: flb_azure_kusto_conf_destroy (azure_kusto_conf.c:628)
==53772==    by 0x22F0EA: cb_azure_kusto_exit (azure_kusto.c:452)
==53772==    by 0x18CF59: flb_output_exit (flb_output.c:315)
==53772==    by 0x1A0661: flb_engine_shutdown (flb_engine.c:917)
==53772==    by 0x1A02B2: flb_engine_start (flb_engine.c:823)

In flb_upstream_destroy, the tls struct is referenced, so if it's destroyed before calling upstream_destory we get this warning. This is caused by the upstream_node and the upstream using the same tls struct

Edit: just noticed my commit message suggests the tls is being freed twice, which is not true, I'll update the message as soon as I push fixes for the rest of your review comments

papigers avatar Jul 08 '22 14:07 papigers

Thanks for the comments you've addressed so far, I'm taking a lunch break and I'll review your changes afterwards. I wanted to mention that I stopped commenting on the initialization / goto instances at some point so I'd advice to take a quick look on your own as that could be a problem.

leonardo-albertovich avatar Jul 08 '22 16:07 leonardo-albertovich

@patrick-stephens Created a documentation PR: fluent/fluent-bit-docs#841

I'll try to finish addressing all the CR comments this week

papigers avatar Jul 11 '22 16:07 papigers

Sure @papigers, @leonardo-albertovich can sign off once he's happy although this will go into 2.0 now unless you want to backport for a 1.9 release (target that branch - but I'd wait for this to be approved first).

Are you ok to do a demo at the next community meeting (28th July I think)? Feel free to ping me direct on Slack or pat-at-calyptia-dot-com.

patrick-stephens avatar Jul 15 '22 12:07 patrick-stephens

@patrick-stephens I'd love if it could get into 1.9, but that's up to you, 2.0 would be fine too, once everything is approved, I guess we'll see :)

As for the demo, I'd be happy to.

papigers avatar Jul 15 '22 16:07 papigers

@leonardo-albertovich @patrick-stephens so should I target 1.9 or leave it as is?

Also, I've noticed that the macos test check is running for over 5 hours? Is that normal?

papigers avatar Jul 16 '22 17:07 papigers

Also, I've noticed that the macos test check is running for over 5 hours? Is that normal?

I don't think any of these taking hours is normal, did you add unit tests for your plugin? we can probably restart it and if the issue persists maybe you can rebase from master to see if it's something that has already been fixed.

As for 1.9 I think we need to check that with @edsiper since last time, when 1.9 was master and 1.8 stable the policy was to only make fixes and important improvements to 1.8 but not add features.

leonardo-albertovich avatar Jul 16 '22 18:07 leonardo-albertovich

I see that there are some commits related to build fixes, so I'll rebase from master/1.9 and give it a go

papigers avatar Jul 16 '22 19:07 papigers

@leonardo-albertovich can you re-approve so it will trigger the checks? 🙏

papigers avatar Jul 18 '22 15:07 papigers

@papigers my opinion is target master for now and once approved we can potentially backport

patrick-stephens avatar Jul 20 '22 16:07 patrick-stephens

@patrick-stephens any idea why would the macos runs run this long and reach a timeout? I'm not sure what needs to be done to fix it

I see that other PRs share the same state, does this block merging?

papigers avatar Jul 21 '22 19:07 papigers

I think there's a fix in master for that macos test issue, I'd suggest rebasing.

leonardo-albertovich avatar Jul 21 '22 20:07 leonardo-albertovich

not sure what this failure is all about, seems flaky, can you confirm?

papigers avatar Jul 22 '22 16:07 papigers

not sure what this failure is all about, seems flaky, can you confirm?

Do you mean the unit test one? It seems to be failing on IPv6 networking.

patrick-stephens avatar Jul 25 '22 09:07 patrick-stephens

Yeah, but I just don't see how it's related to my PR

papigers avatar Jul 25 '22 12:07 papigers

@patrick-stephens @leonardo-albertovich can someone help with completing this PR? if there's a thing that needs fixing I'll be glad to do so but I don't understand how my changes would cause this UT to fail specifically on ubuntu gcc :/

papigers avatar Jul 29 '22 18:07 papigers

I don't think any of the CI issues are relevant to this PR. From my point of view this is mergeable.

leonardo-albertovich avatar Jul 30 '22 10:07 leonardo-albertovich

I don't think any of the CI issues are relevant to this PR. From my point of view this is mergeable.

@patrick-stephens ?

papigers avatar Aug 04 '22 23:08 papigers