[proposal] metric_cache 存储模块设计及重构

[jasonliu747]

What is your proposal: 包括底层持久化，指标存储架构，其余细节待补充

Why is this needed:

Is there a suggested solution, if so, please add it:

[jasonliu747]

/cc @LambdaHJ 辛苦在这下面补充一下你之前的尝试，还有遇到的问题，谢谢～

[LambdaHJ]

将指标使用tsdb存储，一些业务字段需要以label的方式存储到tsdb，如下：

func (ts *tsstorage) InsertPodResourceMetric(n *podResourceMetric) error {
	rows := []tstorage.Row{
		{
			Metric: "pod_resource_cpu",
			Labels: []tstorage.Label{
				{"name", n.PodUID},
			},
			DataPoint: tstorage.DataPoint{
				Value:     n.CPUUsedCores,
				Timestamp: int64(n.Timestamp.Second()),
			},
		},
		{
			Metric: "pod_resource_memory",
			Labels: []tstorage.Label{
				{"name", n.PodUID},
			},
			DataPoint: tstorage.DataPoint{
				Value:     n.MemoryUsedBytes,
				Timestamp: int64(n.Timestamp.Second()),
			},
		},
	}
	for i := range n.GPUs {
		gpuRows := []tstorage.Row{
			{
				Metric: "pod_resource_gpu_memory",
				Labels: []tstorage.Label{
					{"name", n.PodUID},
					{"DeviceUUID", n.GPUs[i].DeviceUUID},
					{"Minor", strconv.Itoa(int(n.GPUs[i].Minor))},
				},
				DataPoint: tstorage.DataPoint{
					Value:     n.GPUs[i].MemoryUsed,
					Timestamp: int64(n.Timestamp.Second()),
				},
			},
			{
				Metric: "pod_resource_gpu_total_memory",
				Labels: []tstorage.Label{
					{"name", n.PodUID},
					{"deviceUUID", n.GPUs[i].DeviceUUID},
					{"Minor", strconv.Itoa(int(n.GPUs[i].Minor))},
				},
				DataPoint: tstorage.DataPoint{
					Value:     n.GPUs[i].MemoryUsed,
					Timestamp: int64(n.Timestamp.Second()),
				},
			},
			{
				Metric: "pod_resource_gpu_smutil",
				Labels: []tstorage.Label{
					{"name", n.PodUID},
					{"deviceUUID", n.GPUs[i].DeviceUUID},
					{"Minor", strconv.Itoa(int(n.GPUs[i].Minor))},
				},
				DataPoint: tstorage.DataPoint{
					Value:     n.GPUs[i].MemoryUsed,
					Timestamp: int64(n.Timestamp.Second()),
				},
			},
		}
		rows = append(rows, gpuRows...)
	}
	return ts.db.InsertRows(rows)
}

这就要求查询metrics时返回数据要包含label。 但是使用到的嵌入式tsdb查询返回不包含label。 https://github.com/nakabonne/tstorage/issues/36

[zwzhang0107]

可以考虑用prometheus的tsdb库 https://github.com/prometheus/prometheus/tree/main/tsdb

[LambdaHJ]

prometheus的tsdb必须要写磁盘。 可能需要挂载hostpath.

[LambdaHJ]

影响模块：metriccache 主要变更代码： storage.go storage_tables.go 副作用：可能会影响使用metriccache 方案： 使用prometheus tsdb模块存储数据。 由于tsdb一定需要写磁盘，规划挂载emptydir解决磁盘读写问题。

[zwzhang0107]

Resource Consumption Comparation: before & after pod & container throttled ratio saved to tsdb.

We deploy koordlet v1.2 and latest version with tsdb refactor on the same node and start 60 pods on it for data collecting.

In v1.2, only latest 30min metrics are saved in sqlite, in latest version we set the time range is extended to 24h.

so we set the default time range as 12h now, in future, we should consider save corse-grained metrics for earlier metrics.