qryn
qryn copied to clipboard
Slow log request performance
log format:
{"body":"body","traceid":"1","spanid":"1","severity":"Information","flags":1,"attributes":{"AccountId":1,"ActionId":"UUID","ActionName":"Controller","Stat":"1","Id":"2","ConnectionId":"123","GId":0,"ParentId":"UUID","RequestId":"UUID","RequestPath":"/path","SpanId":"1323aff603f45021","TraceId":"03016d03700e03ec8b2892038bacfd92","{OriginalFormat}":"INIT"},"resources":{"deployment.environment":"Stage","service.instance.id":"UUID","service.name":"Service","team":"Team","telemetry.sdk.language":"dotnet","telemetry.sdk.name":"opentelemetry","telemetry.sdk.version":"1.9.0"},"instrumentation_scope":{"name":"Controller"}}
Loki request:
{level=~"WARN|INFO|ERROR|FATAL", job="Service"} |= 85577686
| json body="body", Environment="resources["deployment.environment"]", Scope="instrumentation_scope["name"]" | Environment = Stage
Took more then 30 seconds and crush then.
{level=~"WARN|INFO|ERROR|FATAL", job="Service"}
works fine.
{level=~"WARN|INFO|ERROR|FATAL", job="Service"} | json body="body"
7 days gap -> 15.2 seconds response time, "Logs volume" in grafana 502
{level=~"WARN|INFO|ERROR|FATAL", job="Service"} | json body="body", Environment="resources[\"deployment.environment\"]", Scope="instrumentation_scope[\"name\"]" | Environment =
Stage``
7 days gap -> 30 seconds response time -> 502 Client.Timeout exceeded while awaiting headers
Any chance to speed it up?