Issue with log rotation and getting request payload size exceeds the limit

[rahulracker7539]

Describe the bug

Hi Everyone, I am using fluentd as a sidecar container in gke cluster to get the application logs send to google cloud logging coz by default application is not sending logs to stdout/stderr.

here is my configmap gist :-

https://gist.github.com/rahulracker7539/f05e4e03640b42cfd8989656452e6865

here is my sample deployment file gist :- https://gist.github.com/rahulracker7539/672d10d64fd5834acaf727644495f9ed

when we deploying and doing the load test then am seeing the below error msg in logs after some time of starting the load test :-

    **Request payload size exceeds the limit: 10485760 bytes**
    

I already set the log rotation in configmap as per below doc which holds the fluentd configuration but still am seeing the above msg. https://docs.fluentd.org/deployment/logging#log-rotation-setting

I also exec inside the main container and logging agent container and below is the size of file which I can see which probably hints that log rotation is not happening properly.

from main application container

# ls -lh application.log
-rw-r--r--    1 test gir       703.1M Jun 18 09:28 application.log

from logging-agent container

$ ls -lh application.log
total 1.9G
-rw-r--r-- 1   1000   2000 1.9G Jun 18 09:36 application.log

To Reproduce

I have mentioned the above gist url, just need to deploy the configmap and sample deployment and do so load test to generate lots of logs

Expected behavior

It should show the below msg in log when log rotation is already in place.

Request payload size exceeds the limit: 10485760 bytes

Your Environment

- Fluentd version: 1.16.4
- TD Agent version: 
- Operating system:
- Kernel version:

Your Configuration

I have added the configmap above which holds the fluentd.confg

Your Error Log

**Request payload size exceeds the limit: 10485760 bytes**

Additional context

No response