store parquet file in local and then read from disk.

[zdyj3170101136]

i test two type of query: the first query from s3 directly:

func NewBackendReaderAt(ctx context.Context, r objstore.Bucket, name string) *BackendReaderAt {
	return &BackendReaderAt{ctx, r, name, atomic.Uint64{}}
}

func (b *BackendReaderAt) ReadAt(p []byte, off int64) (int, error) {
	b.bytesRead.Add(uint64(len(p)))
	r, err := b.r.GetRange(b.ctx, b.name, int64(uint64(off)), int64(len(p)))
	if err != nil {
		panic(err)
		return 0, err
	}
	n, err := io.ReadFull(r, p)
	if err != nil {
		panic(err)
		return 0, err
	}
	return n, nil
}

func (b *BackendReaderAt) BytesRead() uint64 {
	return b.bytesRead.Load()
}

the second fetch data from s3 and store it to disk, and then read from disk.

both read with options:

o := []parquet.FileOption{
		parquet.SkipBloomFilters(true),
		parquet.SkipPageIndex(true),
		parquet.FileReadMode(parquet.ReadModeAsync),
		parquet.ReadBufferSize(1024 * 1024),
	}

the first cost 4 min. the second cost 1.6s to fetch data, and 6s to query data. the parquet file is 130MB。

so i purpose store parquet file in local and then read from disk.

[mapno]

Hi @zdyj3170101136. Can you explain a bit more what you're asking/suggesting here? I'm not sure I understand what you mean.

While local storage might have some advantages, it's not suitable for distributed deployments—unless all components have access to the same disk, which is rarely the case. You can read about storage in the docs.

[zdyj3170101136]

你好@zdyj3170101136。您能在这里解释一下您的要求/建议吗？我不确定我明白你的意思。

虽然本地存储可能有一些优势，但它不适合分布式部署，除非所有组件都可以访问同一磁盘，但这种情况很少见。您可以在文档中阅读有关存储的内容。

fetch the file from s3 and cache in local. otherwise if we read directly from s3, there would be too much small range read.

[mapno]

fetch the file from s3 and cache in local. otherwise if we read directly from s3, there would be too much small range read.

That sounds simple, but it's no small change. Queriers are designed to be stateless and elastic, so the read path can be scaled up very quickly for demanding queries. Adding PVCs to queriers would require a big refactor of the read path.

There are a lot of considerations needed for this change. Could you give us more details? How would you introduce those disks? Which component would do the maintenance of that data? Maybe another component in the read path?

Not meaning to be dismissive, but I think more thought it's needed for this change. Right now it's not entirely clear to me why the current approach is worse or better than a different one.

[joe-elliott]

the first cost 4 min.

4 mins is a hilariously long time to search a 130MB block. We normally hit TB/s with Tempo internally. This is due to a misconfiguration or a very slow s3 backend. Please share details which help us understand what is happening this case. How many read ranges did Tempo attempt on this block? What was your query? What was the p99/p90/p50 latency of the object storage queries? What is your config?

From this distance with no details it's impossible to diagnose your issue, but 4 mins is several orders of magnitude longer than we see internally.

[zdyj3170101136]

第一个花费 4 分钟。

对于搜索 130MB 的块来说，4 分钟是一个非常长的时间。我们通常在内部使用 Tempo 达到 TB/s。这是由于配置错误或 s3 后端非常慢造成的。请分享详细信息，以帮助我们了解此案例的情况。Tempo 在此块上尝试了多少个读取范围？您的查询是什么？对象存储查询的 p99/p90/p50 延迟是多少？你的配置是什么？

从这个距离来看，没有详细信息，不可能诊断您的问题，但 4 分钟比我们内部看到的时间长了几个数量级。

18533 s3 get range query, fetch data total cost 317 seconds, total fetch 140 MB.

it's a parquet file i created for my project: encoding with gzip, 1024 row group.

type Meta struct {
	Name        string
	PeriodType  string
	PeriodUnit  string
	SampleType  string
	SampleUnit  string
	Timestamp   int64 `parquet:",timestamp"` // time.Time is not supported
	Duration    int64
	Period      int64
	ProfileID   string
	LabelsKey   []string `parquet:",list"` // this value definition level is always 1
	LabelsValue []string `parquet:",list"`
	TotalValue  int64    `parquet:","`
	SampleLen   int64    `parquet:","`
}

[joe-elliott]

This is a completely different schema that is being queried with unknown code. The analysis you've performed has nothing to do with Tempo.

[zdyj3170101136]

这是一个完全不同的模式，正在使用未知代码进行查询。您执行的分析与 Tempo 无关。

yes, but it testify that sometimes readfrom parquet require lots of range query.

and this is very slow to query directly from s3.

[mazikai002]

want to add VX of zdyk3170101136

[zdyj3170101136]

want to add VX of zdyk3170101136

18888921036

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.