qryn icon indicating copy to clipboard operation
qryn copied to clipboard

Slow log request performance

Open deathalt opened this issue 7 months ago • 13 comments

log format:

{"body":"body","traceid":"1","spanid":"1","severity":"Information","flags":1,"attributes":{"AccountId":1,"ActionId":"UUID","ActionName":"Controller","Stat":"1","Id":"2","ConnectionId":"123","GId":0,"ParentId":"UUID","RequestId":"UUID","RequestPath":"/path","SpanId":"1323aff603f45021","TraceId":"03016d03700e03ec8b2892038bacfd92","{OriginalFormat}":"INIT"},"resources":{"deployment.environment":"Stage","service.instance.id":"UUID","service.name":"Service","team":"Team","telemetry.sdk.language":"dotnet","telemetry.sdk.name":"opentelemetry","telemetry.sdk.version":"1.9.0"},"instrumentation_scope":{"name":"Controller"}}

Loki request:

{level=~"WARN|INFO|ERROR|FATAL", job="Service"} |= 85577686 | json body="body", Environment="resources["deployment.environment"]", Scope="instrumentation_scope["name"]" | Environment = Stage

Took more then 30 seconds and crush then.

{level=~"WARN|INFO|ERROR|FATAL", job="Service"}

works fine.

{level=~"WARN|INFO|ERROR|FATAL", job="Service"} | json body="body"

7 days gap -> 15.2 seconds response time, "Logs volume" in grafana 502

{level=~"WARN|INFO|ERROR|FATAL", job="Service"} | json body="body", Environment="resources[\"deployment.environment\"]", Scope="instrumentation_scope[\"name\"]" | Environment = Stage``

7 days gap -> 30 seconds response time -> 502 Client.Timeout exceeded while awaiting headers

Any chance to speed it up?

deathalt avatar Jul 24 '24 13:07 deathalt