Dan Goodman

Here are some updated logs, seems like some of the JSON logs are the pods we don't control in EKS :P ``` 2022-03-29T18:27:59.052628Z INFO transform{component_kind="transform" component_id=setlevel component_type=remap component_name=setlevel}: vrl_stdlib::log: "failed...

This has just started happening again, no error or warning logs. Anything else I should be looking for or trying? We can't be dropping our production logs.

@jszwedko one thing I am noticing is that we don't get the logs about vector finding new files. This is easier to reproduce on things that come and go relatively...

@wtchangdm will try, but it’s the case for long running pods too where it just stops recording logs for those pods. Those files definitely stay around longer than 60s

Interesting, but we don’t get any warnings or errors like that does.

@jszwedko I will say that so far we haven't _noticed_ any issues since we dropped the glob cooldown to `8000`, but we also surely have not been in our logs...

@jszwedko This went very not well, was printing this extremely quickly: ``` 2022-04-25T18:40:35.870916Z WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=InitialListFailed(Api(ErrorResponse { status: "Failure", message: "pods is forbid den:...

~Ah ok I needed to update the cluster role tags too :)~ still getting it...

yep, just made that change :)

After sending ~50log/s I can see the following errors show, then logs stop shipping. It prints a ton of the error logs. It is able to ship them for about...