Jörn Friedrich Dreyer
Jörn Friedrich Dreyer
hm and autoaccept shares in the frontend is also too slow to consume the events: ```diff diff --git a/services/frontend/pkg/command/events.go b/services/frontend/pkg/command/events.go index 2487165533..954fab7af8 100644 --- a/services/frontend/pkg/command/events.go +++ b/services/frontend/pkg/command/events.go @@ -118,24 +118,26...
hm restarting the userlog service quiets it ... restarting the fronted seems to pick up events again
testing 6.3.0 ``` # k6 run ~/cdperf/packages/k6-tests/artifacts/koko-platform-000-mixed-ramping-k6.js /\ |‾‾| /‾‾/ /‾‾/ /\ / \ | |/ / / / / \/ \ | ( / ‾‾\ / \ | |\...
after making userlog and frontend more concurrent still broken -> `http_req_failed................: 35.86% ✓ 1662 ✗ 2972` ``` # k6 run ~/cdperf/packages/k6-tests/artifacts/koko-platform-000-mixed-ramping-k6.js /\ |‾‾| /‾‾/ /‾‾/ /\ / \ | |/...
rerunning after things settled down -> getting worse: `http_req_failed................: 52.12% ✓ 2110 ✗ 1938` ``` # k6 run ~/cdperf/packages/k6-tests/artifacts/koko-platform-000-mixed-ramping-k6.js /\ |‾‾| /‾‾/ /‾‾/ /\ / \ | |/ / /...
and another run after things settled down ... getting worse `http_req_failed................: 80.37% ✓ 2936 ✗ 717` ``` k6 run ~/cdperf/packages/k6-tests/artifacts/koko-platform-000-mixed-ramping-k6.js /\ |‾‾| /‾‾/ /‾‾/ /\ / \ | |/ /...
hm ... registry is still not forgetting nodes: ``` │ proxy-7cf6ffcfc9-2jgq7 {"level":"error","service":"proxy","request-id":"5daa8746-6bf7-4e93-b7c0-88ee2696cb0c","userid":"afabfc40-6cdc-4444-8c15-da201b23d46e","error":"{\"id\":\"go.micro.client\",\"code\":503,\"d │ │ etail\":\"connection error: desc = \\\"transport: Error while dialing: dial tcp 10.96.8.35:9191: i/o timeout\\\"\",\"status\":\"Service Unavailable\"}","time":"2024-09-12T13:13:01Z","line":"github.com/owncloud/o │ │...
hm so this is a case where we try to make a connection too early. the gatewoay somtimes cannot connet to the storage users service: ``` "[pod/gateway-754d8b486c-x7p6z/gateway] {\"level\":\"error\",\"service\":\"gateway\",\"pkg\":\"rgrpc\",\"traceid\":\"8d57bf5baf9b67274185cb25ee09a891\",\"user-agent\":\"k6/0.45.0 (https://k6.io/)\",\"from\":\"tcp://10.96.17.28:59456\",\"uri\":\"/cs3.gateway.v1beta1.GatewayAPI/ListContainer\",\"start\":\"12/Sep/2024:16:34:39 +0000\",\"end\":\"12/Sep/2024:16:34:39...
I have this 99x in the log: ``` "[pod/storagesystem-c9d9b459d-fvjb4/storagesystem] {\"level\":\"warn\",\"service\":\"storage-system\",\"pkg\":\"rgrpc\",\"traceid\":\"8c8e5026ceff9a43febe0825e19d5638\",\"error\":\"error parsing token: Token used before issued\",\"time\":\"2024-09-12T19:40:16Z\",\"line\":\"github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/auth/auth.go:150\",\"message\":\"access token is invalid\"}" ``` WTF ... clock not in sync? How do we handle...
and a panic in `[pod/gateway-6fddd795b-b8njq/gateway]`: ``` { "level": "error", "service": "gateway", "pkg": "rgrpc", "traceid": "51fc70040720c8b5e2e865285d7af23a", "time": "2024-09-12T21:31:10Z", "line": "github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/recovery/recovery.go:50", "message": "user not found in context; stack: goroutine 1381306 [running]: runtime/debug.Stack()...