Support running Tempo distributed without object store

[yvrhdn]

Is your feature request related to a problem? Please describe.

It's sometimes impractical or undesirable to run Tempo with an object store. There should be a mechanism to run Tempo in microservices mode with ingester-local storage.

It's currently 'possible' to run Tempo in microservices mode without remote backend (you can configure backend: local) and data will simply be stored in the ingesters. There are some limitations with this approach:

• Data is not queryable after the block is flushed: flushed blocks will still be stored on the ingester (just not in the WAL anymore) and the querier will not be able to query these blocks.
• It's hard to control disk usage of the ingester: existing configuration options are all time-based ("remove a block after x hours"). If you store data on the ingester it would be nice to be able to say "remove the oldest block if total storage size is >100GB".

Describe the solution you'd like

Provide a configuration to run Tempo without backend (or with a "local-distributed" backend). In this mode data is stored on the ingester and remains queryable until the block is completely removed. Queriers don't try to query the backend but only issue requests to the ingesters.

Provide an option to control block retention in the ingester based upon storage size, not only age.

Describe alternatives you've considered

You can do a hacky workaround right now by configuring the local backend and increasing complete_block_timeout. This will keep the blocks in the ingester WAL for longer. This is really not ideal as flushed blocks can not be reached by the queriers and controls are a bit crude.

Additional context

[joe-elliott]

Honestly, if someone wanted this then treating complete_block_timeout as total retention would work as well as anything we put together. In fact, it would be functionally identical to writing the code to have ingesters search some kind of local store that we flushed blocks too.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.