Support a directory as input, with subdirectories

[spellr]

Is your feature request related to a problem? Please describe.

The problem is that it's not possible to set a entire directory in INPUT, and get all the logs in the directory including subdirectories.

Using the tail plugin, I can set:

[INPUT]
    Name      tail
    Path      /var/log/*

But it will only get the files immediately under /var/log

Describe the solution you'd like

The way to achieve this in FluentD is to use **, which matches recursively subdirectories. We can have the same solution here.

Describe alternatives you've considered

Maybe a flag recursive=True, but I guess it's less intuitive and harder to code.

Additional context

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[dsalvet]

bump

[vnedyavila]

bump

What is the status here? This seems like a very important feature

[servo1x]

This would be a super useful feature.

[sujairamprasathc]

Any ideas about on which version of fluent-bit can we expect this feature implementation...

[rohit-mundayadan]

I would love to have this feature. I have a web server which hosts 100s of web apps/services (kind of microservice but not containers). It would be hard to configure and maintain 100s of paths (and removing and adding paths as services are added/removed) vs just having to point to the root of the web servers hosting path and get all log files in subdirectories.

[JasonMars]

I love the feature very much. I maintain a server which has multiple daemons to write logs to certain directory, but on different sub directories. It would be very useful if the feature gets implemented which would help the configuration as clean as possible.

[HarleyB123]

Having this feature would be great! There isn't really a workaround (that I can think of) for this, so it's blocking me from using fluentbit

[stevenolen]

I'm not sure this will help out in any generic case, but perhaps help to those who have been looking for this feature! I recently had a use-case where I needed to collect the json-file logging driver output from the docker daemon and stumbled on this solution for fluent-bit that does indeed tail each container log file matching this pattern: /var/lib/docker/containers/<container_id>/<container_id>-json.logs

    [INPUT]
        Name             tail
        Path             /var/lib/docker/containers/*/*-json.log
        ...

[da-head0]

@stevenolen That case only suit for specific case..

    [INPUT]
        Name             tail
        Path             /a/b/c/d/*/*.log
        ...

only find files in path such as /a/b/c/d/e/*.log , not for /a/b/c/d/e/f/g/*.log

[izeal]

We still need this feature

[mfolivas]

bump

[jhaldar22]

bump

[linczy]

+1 vote for this feature. One option to create symlinks and the symlink files can be in one directory.

[SubhraB]

+1

[umialpha]

+1

[wiperpaul]

+1

[yjagdale]

+1

[skiedude]

+1

[Shreeraj-D]

+1

[YikaiHu]

+1

[barabasakos]

+1

[air3ijai]

It is supported in Vector File source with globbing

[sources.my_source_id]
type = "file"
include = [ "/var/log/**/*.log" ]

[rairajiv-trellix]

Any update, please?

[johnmmcgee]

Would also like.

[conawolenr]

+1

[blchilders]

+1

[cazorla19]

+1

[murrayconnorstewart]

+1