cortex icon indicating copy to clipboard operation
cortex copied to clipboard

Error "out of bounds" after uploading data from future

Open kojderszymon opened this issue 2 years ago • 1 comments

Describe the bug When data is sent (via remote_write) from the future to the Cortex (e.g., data two hours later or more), the Cortex stops accepting any current data. Errors appear in the logs: "err: out of bounds". This is most likely related to the inability to upload data older than an hour back (after uploading data from 2 hours ago, the current data is too old compared to the latest sample).

Questions

  • Is there a way around the problem by uploading data from the future as another tenant or something like that?
  • Is there a setting that will help me upload future data without restrictions?
  • Is there a chance that this will be fixed in the future?

kojderszymon avatar Sep 16 '22 10:09 kojderszymon

That's because currently we can only append data to the tsdb head chunk.

One thing you can try is to keep the head chunks open for more time tweaking this config

  # TSDB blocks range period.
  # CLI flag: -blocks-storage.tsdb.block-ranges-period
  [block_ranges_period: <list of duration> | default = 2h0m0s]

alanprot avatar Sep 20 '22 16:09 alanprot

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Jan 07 '23 17:01 stale[bot]

I think out of bounds is more related to data being too old, not about data in the future. This issue is a duplicate of https://github.com/cortexproject/cortex/issues/2366

friedrichg avatar Mar 08 '23 09:03 friedrichg