fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard

bin: config: Provide maxstdio option for increasing I/O limit on Windows

Open cosmo0920 opened this issue 1 year ago • 3 comments

With default limit of I/O on Windows is somewhat small for large amount of log volume environment. I provide an option to set up this limit up to 2048.


Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

  • [ ] Example configuration file for the change
  • [ ] Debug log output from testing the change
  • [ ] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • [ ] Run local packaging test showing all targets (including any new ones) build.
  • [ ] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • [x] Documentation required for this feature

https://github.com/fluent/fluent-bit-docs/pull/1528

Backporting

  • [ ] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

cosmo0920 avatar Dec 11 '24 10:12 cosmo0920

@cosmo0920 we don't do ulimit stuff in Fluent Bit, why should we do it for Windows ? (maybe we just need to update the docs for windows users ?)

edsiper avatar Dec 19 '24 20:12 edsiper

Is this limit something that can actually be changed at a system configuration level on Windows?

This is the MS doco about the setter function: https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/setmaxstdio?view=msvc-170 . It seems to be specifically something that's part of the Microsoft C runtime library (CRT).

There's some info here saying that the limit can be further stretched if you use specific Win32 calls directly rather than this C runtime library: https://stackoverflow.com/questions/870173/is-there-a-limit-on-number-of-open-files-in-windows/4276338#4276338

nuclearpidgeon avatar Dec 20 '24 01:12 nuclearpidgeon

@cosmo0920 we don't do ulimit stuff in Fluent Bit, why should we do it for Windows ? (maybe we just need to update the docs for windows users ?)

In Windows, we need to set up the ulimit like limit via _setmaxstdio. There is no API/mechanism without this API from C runtime.

cosmo0920 avatar Dec 20 '24 01:12 cosmo0920

thanks folks for the context. @cosmo0920 I suggest to rename this option to windows.maxstdio

edsiper avatar Dec 29 '24 02:12 edsiper

Sure, I'll rename it from the previous one.

cosmo0920 avatar Dec 29 '24 12:12 cosmo0920

I rebased off the current master and added windows. prefix on the newly introduced parameter.

cosmo0920 avatar Dec 30 '24 14:12 cosmo0920

will merge after rebasing to avoid conflict.

edsiper avatar Mar 29 '25 16:03 edsiper

I rebased off the current master.

cosmo0920 avatar Apr 29 '25 21:04 cosmo0920

@cosmo0920 for this one please make sure to submit documentation updates.

edsiper avatar May 08 '25 14:05 edsiper