foundationdb icon indicating copy to clipboard operation
foundationdb copied to clipboard

txnCommitErrors should not include "illegal_tenant_access"

Open jzhou77 opened this issue 3 years ago • 2 comments

In status json, we report cluster.workload.operations.memory_errors to measure the number of requests rejected by the proxies because the memory limit has been exceeded, which includes TxnCommitErrors, KeyServerLocationErrors, and TxnRequestErrors. All of them are based on memory usage.

So I think the code here: https://github.com/apple/foundationdb/blob/c4c6aebde5cc2fe90f2adb6d2ad0eb4e18d0856b/fdbserver/CommitProxyServer.actor.cpp#L398-L402 should be modified to not counting tenant errors as part of txnCommitErrors.

jzhou77 avatar Aug 10 '22 20:08 jzhou77

I wonder if any of these fields were intended to be scoped to memory errors, or if that's just the only type of errors that were applicable at the time. If these are intended to be memory scoped, we could rename them, or if not we could add a separate counter for memory errors.

sfc-gh-abeamon avatar Aug 10 '22 21:08 sfc-gh-abeamon

@sfc-gh-svemuri I think it makes sense to leave this line here, but maybe what we should do is add a separate variable to track memory errors and report that in status (assuming what @jzhou77 really wants is to track memory errors). We could also optionally report overall commit, key server location, or GRV request errors in status too, but I'm not sure we need to do that now.

sfc-gh-abeamon avatar Aug 16 '22 15:08 sfc-gh-abeamon