tempo icon indicating copy to clipboard operation
tempo copied to clipboard

Block start can be set before block end when sending older spans

Open joe-elliott opened this issue 1 year ago • 1 comments

Describe the bug While working on configuration options to allow for several hour old spans to be ingested @critchtionary discovered that the block end time can sometimes be set before the block start time.

@critchtionary, if you are able to provide any details on reproducing this it would be appreciated.

joe-elliott avatar Nov 30 '23 17:11 joe-elliott

Reproducer using a fresh deployment with default config (ingestion_time_range_slack=2m and query_ingesters_until=30m):

  1. Send in a trace with end time older than query_ingesters_until e.g. 31 minutes ago.
  2. Immediately send in another trace with end time older than query_ingesters_until + ingestion_time_range_slack e.g. 34 minutes ago.
  3. Look at meta.json in /var/tempo/wal/ - end time will be before start time:
"startTime":"2023-12-01T09:17:08Z",
"endTime":"2023-12-01T08:45:59Z"

critchtionary avatar Dec 01 '23 09:12 critchtionary