fluent-bit
fluent-bit copied to clipboard
fluent
Allow customizing the number of outputs
Is your feature request related to a problem? Please describe.
There is currently a hard limit on the number of outputs that can be configured.
This limit is not documented at the moment, there is an issue for that.
The current landing page of fluentbit has a statement on scalability stating: "1pb Data Throughput Across Thousands Of Sources And Destinations Daily". And: Dynamic Routing ... Distribute data to multiple destinations with a zero copy strategy I misinterpreted this to fit my use-case.
Describe the solution you'd like
A configuration that allows to change this limit.
Describe alternatives you've considered
Fluentd seems to support many more outputs, but I have not tried implementing it as the footprint seems too high.
Some kind of sharding in fluent-bit would allow to keep the number of outputs fixed, and keep the limit. This sharding could be managed by the fluent-operator.
Additional context
We have an on-premise multi-tenant Kubernetes cluster where an API provides a varying endpoint for bulk ingestion in OpenSearch. Each new tenant on the cluster gets a dedicated OpenSearch tenant. We link the tenant in the k8s cluster to the ingestion endpoint using an output plugin. This endpoint is not known in advance. As tenants are self-serve, the amount of outputs might reach in the thousands.
Through a cluster management API, we use the fluent-operator to easily drop ClusterOutput documents to manage the binding from tenant-namespaces to OpenSearch endpoints.
I have not seen other solutions that are as elegant as the plugin system of fluent-bit, maybe there is an equivalent in fluentd, but overhead does matter in a shared cluster setting so that a minimal footprint is a desired quality.
Open question: Allowing the increase of outputs will have a memory impact and other compute side effects. I am leaning on the intuition that the low footprint of fluent-bit allows for more vertical scaling, but I do not know the internals to warrant if the assumption holds.
References:
- #4159 Reference on an issue that points to this current limitation.
- My comment for describing this use-case.
- Similar use-case of namespace templated looping on inputs from @lmuhlha
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Not stale. The use case could be implemented in a kind of unique multi output plugin (instead of multiple outputs just to change the target). To be investigated.
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.
@lecaros any plans for this feature request? Seeing that this has been open for a year, I also have a multi-tenant use case where I'd want 1000+ Outputs configured.
Funny enough, I've been working on something related to routing yesterday and was thinking about improving the routing mask feature, partially because I noticed the same issue (although I wasn't thinking about having more than 256 outputs).
I just took a quick look at your PR @adiforluls and unless I missed something it seems that you didn't update the routes_mask field in flb_input_chunk which flb_routes_mask_set_by_tag modifies which means with the new limit you would be overwriting bits from the chunk list link that follows the route array or adjacent memory.
Honestly, I didn't like the code I saw and I can't dedicate time to properly reviewing it right now but if no one else does, you stick around for about a week and you are willing to make some changes to it I might be able to help you.
@leonardo-albertovich I'm willing to incorporate any changes requested. I went for the smallest possible changes required here since the limit of 256 outputs is a blocker for me. But if there's a cleaner, more robust way I'm happy to go on that route!
Super cool, I'll save that PR and get back to it next week when I have time to properly review it and ask for the necessary changes.
Thanks a lot for the initiative!
@leonardo-albertovich could you review the PR and suggest the necessary changes? Thanks in advance https://github.com/fluent/fluent-bit/pull/7281
Thank you for reminding me of this, I finally got a bit of time so I'll review your PR this afternoon.
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.
This issue was closed because it has been stalled for 5 days with no activity.
still an issue and still desired by multi-tenant workloads (k8s). Has anyone come up with a workaround / hack? The pull request mentioned above seems stalled.
I'll append as well. Multi-tenant use-case here too and indeed troublesome.
