obslytics
obslytics copied to clipboard
thanos-community
Don't know how to use this tool, need enhance documentation
Hi, we are a bunch of data scientists. We are trying to leverage prometheus metrics data to do AIOPS. Metrics data are already stored in AWS S3 via Thanos sidecar approach. To make data consumable by Spark, need a tool to convert data. Looks like this tool can be helpful, but it's really hard to understand how to use this tool given neither of us have Go programming experiences...
For instance, we are testing below commands:
go run ./cmd/obslytics export --input-config='{"endpoint":"127.0.0.1:10901","tls_config":{"insecure_skip_verify":true}}' --match="container_cpu_load_average_10s" --resolution=2h --output-config-file=container_cpu_load_average_10s.parquet --debug --min-time="2021-08-24T20:00:00+00:00" --max-time="2021-08-25T20:00:00+00:00"
The console gets error codes:
level=error ts=2021-08-25T09:32:26.912354Z caller=main.go:104 msg="running command failed" err="unsupported Reader type " exit status 1
No idea why, something related to StoreAPI. We deployed Thanos sidecar and Querier, and then kubectl port-forward querier pod like this: kubectl port-forward thanos-addons-query-7fc47d8cff-tf4rg 10901:10902
Update:
By reading source codes, we made it work finally. First port-forward Thanos StoreAPI, then run command:
go run ./cmd/obslytics export --input-config='{"type":"STOREAPI","endpoint":"127.0.0.1:10901","tls_config":{"insecure_skip_verify":true}}' --output-config='{"type":"parquet", "path":"metrics/up", "storage":{"type":"s3", "config": {"bucket": "xxx", "endpoint": "s3.us-east-1.amazonaws.com", "insecure": true, "access_key": "key", "secret_key": "secret"}}}' --match="up" --resolution=2h --min-time="2021-08-24T23:59:59Z" --max-time="2021-08-26T00:59:59Z" --debug
