opa icon indicating copy to clipboard operation
opa copied to clipboard

default behaviour should be to throw away decision logs instead of running OOM if its unable to send the decision logs

Open johanneslarsson opened this issue 2 years ago • 3 comments

After a discussion on Slack I realised that OPA would end up in a OOM, if its unable to push its decision logs to the control plane. https://openpolicyagent.slack.com/archives/CBR63TK2A/p1651663584624969. I haven't verified this is the case yet.

What part of OPA would you like to see improved?

decision_logs.reporting.buffer_size_limit_bytes should probably have default value that suits the smallest containers. I understand that there are use cases where logs are required, but a crash wouldn't help there either.

Describe the ideal solution

I think most people want OPA to rather throw away logs than causing runtime exceptions.

johanneslarsson avatar May 04 '22 11:05 johanneslarsson

I haven't verified this is the case yet.

We should try to do that. 🔍 I'd imagine an infinite loop that queries, queries, queries, while having the decision logger configured to send to a port without a service, should trigger this...?

srenatus avatar May 04 '22 11:05 srenatus

Hey Johannes 👋😃

Optimizing the value for minimal resource allocation sounds a bit risky to me — I would not want OPA to discard decisons if it's using like 10% of the memory I've allocated for the process. In an ideal scenario you'd be able to do something like:

decision_logs.reporting.buffer_size_limit_available_mem: 80%

But I'm guessing there's no reliable way of doing that.

An alternative could be to dump logs to disk if possible: https://github.com/open-policy-agent/opa/issues/3333

anderseknert avatar May 04 '22 12:05 anderseknert

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days.

stale[bot] avatar Jun 03 '22 14:06 stale[bot]

Closing this in favor of https://github.com/open-policy-agent/opa/issues/4659

ashutosh-narkar avatar Mar 08 '23 02:03 ashutosh-narkar

@ashutosh-narkar the ticket you mentioned as favored is this one?

johanneslarsson avatar Mar 08 '23 06:03 johanneslarsson

Sorry @johanneslarsson. I meant this one. Thanks for catching this.

ashutosh-narkar avatar Mar 08 '23 08:03 ashutosh-narkar