Marcus Teixeira
Marcus Teixeira
@okushchenko, @kelseyhightower
Same error with gcs backend.
note: I even looked at the issue #4713 , but it doesn't seem to be the same context. I applied the changes that were reported, but but were not effective....
> Not being able to flush chunks can definitely lead to memory issues and hitting limits. > > Is there any more context around those failure logs? Could you check...
@slim-bean, @kavirajk Here are more details of what I've been investigating about loki ingester. I changed concurrent_flush to 4 (default value is 16), but I didn't notice any real improvement...
For better troubleshooting experience in this case, I am attaching the pprof file with the flamegraph example. OOM Issue flamegraph: [https://flamegraph.com/share/f1682ecd-8a77-11ec-9068-d2b21641d001]() ![OOM-Issue-flamegraph](https://user-images.githubusercontent.com/40602844/153421234-be5fc274-42c9-4901-9d31-c542173b99f0.png) PS: Issue debug template https://github.com/grafana/loki/issues/5360
@slim-bean FYI > I think we need to figure out what context is getting canceled, i'm not sure there is much you can do to find this other than maybe...
@rsteneteg Thanks for the flush_op_timeout parameter hints. In my environment I had already noticed this detail and it is currently set at 10m. Still, the problem remains :/ When I...
> We use snappy everywhere, gzip compresses more than snappy but is much more cpu expensive at query time so we use snappy for the improved performance. > > how...
@kavirajk Could you put this change in changelog.md? In order to have the tracking of which release this fix was/will be implemented.