Update the max_traces_per_user to be a bytes limit

[joe-elliott]

Describe the solution you'd like Make the final limit to be bytes based instead of trace/span based. The last limits that are based on a count of traces or spans instead of bytes are:

  MaxLocalTracesPerUser  int `yaml:"max_traces_per_user"`
  MaxGlobalTracesPerUser int `yaml:"max_global_traces_per_user"`

[annanay25]

Actually max_traces_per_user together with max_bytes_per_trace does enforce a byte-limit.

Not sure if we wan't to add another or just work with these.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.

[yvrhdn]

A tenant sending a lot of very small traces will trip up max_traces_per_user without actually straining the ingester significantly. In this case we would have to increase max_traces_per_user which can risk blowing up total bytes when they start sending larger traces.

Total bytes is max_traces_per_user * max_bytes_per_trace. We don't have a limit to constrain both together so tenants with different trace patterns would require different tunings.

I propose we add a limit max_live_traces_bytes which enforces total bytes of a batch.

So the 3 limits in the ingester would be:

1. max_bytes_per_trace: constraint how large a trace we allow, we need this limit to prevent huge traces from clogging up Tempo
2. max_traces_per_user: constrain how many unique traces can be added at once, I expect this value can be pretty high
3. max_live_traces_bytes: a combination of both (1) and (2) which protects Tempo against huge batches