ddvk
High xochitl CPU Usage on Syncs Between rM 3.18.2.3 and rmfakecloud v0.0.24
Hardware: rm2 OS: 3.18.2.3 Cloud: rmfakecloud v0.0.24 via Docker in Kubernetes, via /etc/hosts override, no proxy on-device
Symptoms: While connected to wifi, there is significant UI slowness while any sort of sync activity is happening (opening docs, drawer, even changing pens, toolbar). Upon investigation via htop on the tablet, a single xochitl process consumes a full CPU core for several seconds during sync operations (device wake, document closure, etc).
I've tried the following troubleshooting steps in an attempt to isolate the issue, with no changes in symptoms:
- Rebooted tablet
- Restarted rmfakecloud container
- Recreated rmfakecloud user to force a fresh tablet -> cloud sync
I swapped to rM's official cloud to compare performance, and the issue is not present.
Please let me know what other information I can gather to assist.
For another data point, I tried downgrading rmfakecloud to v0.0.23 and that seems to work with no problems, no performance issue on the tablet.
Is there anything in the logs for either the device or rmfakecloud that indicates that there is an endpoint that is not handled?
I faced this situation multiple times during development, on several pre-releases.
In my situation, there was some interesting repeating logs on the tablet, if I remember correctly.
I'm back on rmfakecloud v0.0.24 for testing, and I'm not seeing the issue anymore. The sequence of events this morning:
- Tested on rmfakecloud v0.0.24, issue present (as it has been for several weeks)
- Unpaired, paired with official reMarkable Cloud, performed sync, issue not present.
- Downgraded rmfakecloud to v0.0.23, unpaired tablet from official cloud, paired with downgraded rmfakecloud, issue not present.
- Upgraded rmfakecloud to v0.0.24, issue not present.
I do see this log on the tablet currently:
May 19 16:34:05 reMarkable xochitl[242]: 16:34:05.960 rm.epaperkeyboardhandler No keymap set by QT settings or firmware, defaulting to US.
May 19 16:34:06 reMarkable rm-sync[292]: rm.synchronizer: -- execute requested: 2025-05-19T16:34:06.074
May 19 16:34:06 reMarkable rm-sync[292]: rm.synchronizer: -- running sync, batch: 0/1, minimal:true
May 19 16:34:06 reMarkable rm-sync[292]: rm.synchronizer: document 34c048f2-5c86-4bcb-8a0c-9d4be28da5a6 has failure, not locking
May 19 16:34:06 reMarkable rm-sync[292]: rm.synchronizer: -- execute ended: 2025-05-19T16:34:06.757
I looked up that document, and it is one I deleted during my test swap to rM official cloud, as this document refused to sync.
Looking back in journalctl from a boot where I was having issues (though I don't have particular timestamps for when I noticed 100% CPU usage), I see logs like:
May 19 12:35:46 reMarkable rm-sync[291]: rm.synchronizer: -- execute requested: 2025-05-19T12:35:46.838
May 19 12:35:46 reMarkable rm-sync[291]: rm.synchronizer: -- running sync, batch: 0/1, minimal:false
May 19 12:35:47 reMarkable rm-sync[291]: qt.network.http2: connection error: DATA on invalid stream
May 19 12:35:47 reMarkable rm-sync[291]: qt.network.http2: stream 11 finished with error: "DATA on invalid stream"
May 19 12:35:47 reMarkable rm-sync[291]: qt.network.http2: stream 13 finished with error: "DATA on invalid stream"
May 19 12:35:47 reMarkable rm-sync[291]: rm.network.qt: Received status 200 OK, but Qt Error code 499
May 19 12:35:47 reMarkable rm-sync[291]: rm.network.qt: RequestId=343: Received QNetworkReply error code 499
May 19 12:35:47 reMarkable rm-sync[291]: rm.network.transaction: Request failed: RequestId=343 error='Error during response receipt' http=none url=https://internal.cloud.remarkable.com/sync/v3/files/9567634f6ecd9f10924533b789330bb36b089cf8eb45e91b27724785249482dd
May 19 12:35:47 reMarkable rm-sync[291]: rm.network.transaction: RequestId=343: transaction ended in 412ms after following events:
May 19 12:35:47 reMarkable rm-sync[291]: Create 2025-05-19 12:35:47.551034834
May 19 12:35:47 reMarkable rm-sync[291]: Send 2025-05-19 12:35:47.552334540
May 19 12:35:47 reMarkable rm-sync[291]: Fail 2025-05-19 12:35:47.963558393
May 19 12:35:47 reMarkable rm-sync[291]: rm.network.qt: RequestId=344: Received QNetworkReply error code 499
May 19 12:35:47 reMarkable rm-sync[291]: rm.synchronizer: Communication error when transferring: Not received, error: Unspecified network error
May 19 12:35:47 reMarkable rm-sync[291]: rm.synchronizer: -> failing rmsync::UploadBlobJob for 05f94a20-01d0-47e8-9fb4-c9dc3f2f6e1b.content with RequestNetworkError, 0, ""
May 19 12:35:47 reMarkable rm-sync[291]: rm.synchronizer: - onUploadBlobFailed(): path: 05f94a20-01d0-47e8-9fb4-c9dc3f2f6e1b.content, code: RequestNetworkError, domainError: 0 job:8
May 19 12:35:47 reMarkable rm-sync[291]: rm.synchronizer: network job failed: path: 05f94a20-01d0-47e8-9fb4-c9dc3f2f6e1b.content, code: RequestNetworkError, domainError: 0 job:8, availability=true
May 19 12:35:47 reMarkable rm-sync[291]: rm.synchronizer: requesting abort and cleanup
May 19 12:35:47 reMarkable rm-sync[291]: rm.synchronizer: requesting abort and cleanup
May 19 12:35:47 reMarkable rm-sync[291]: rm.network.transaction: Request failed: RequestId=344 error='Unspecified network error' http=none url=https://internal.cloud.remarkable.com/sync/v3/files/d947af42e3daf5f8d3b4eda5c9fafc3fdd4f1a3df62704102545e1cb1a2aabea
May 19 12:35:47 reMarkable rm-sync[291]: rm.network.transaction: RequestId=344: transaction ended in 414ms after following events:
May 19 12:35:47 reMarkable rm-sync[291]: Create 2025-05-19 12:35:47.558305961
May 19 12:35:47 reMarkable rm-sync[291]: Send 2025-05-19 12:35:47.558889191
May 19 12:35:47 reMarkable rm-sync[291]: Fail 2025-05-19 12:35:47.973002446
May 19 12:35:47 reMarkable rm-sync[291]: rm.synchronizer: Abort called in state State_TransfersAndRemoval
May 19 12:35:47 reMarkable rm-sync[291]: rm.network.http.manager: Tried to abort non-existing transaction id RequestId=343
May 19 12:35:47 reMarkable rm-sync[291]: rm.network.http.manager: Tried to abort non-existing transaction id RequestId=343
May 19 12:35:48 reMarkable rm-sync[291]: rm.synchronizer: -- execute ended: 2025-05-19T12:35:48.093
I don't see anything related to the troublesome document 34c048f2-5c86-4bcb-8a0c-9d4be28da5a6 in those logs however.
@nemunaire during your pre-release testing, do you recall if issues arose due to individual documents?
I will stay on v0.0.24 and try to gather more information if the issue reoccurs. Since I'm no longer able to replicate on v0.0.24, I'm also no longer confident that .23 vs .24 makes any difference.
Hardware: Remarkable Paper Pro
OS: 3.18.2.3
Cloud: ddvk/rmfakecloud:0.0.24 container image
I am using the on-device proxy (v0.0.6).
In my case, when I see the usage, I will see xochitl use up an entire core for a period of time, and then frequently see rmfakecloud-proxy use a full core, briefly.
When I see usage spikes, I've got messages in the logs that look like:
May 20 20:50:26 imx8mm-ferrari xochitl[10890]: 20:50:25.321 rm.network.qt RequestId=10730: Received QNetworkReply error code 5 (calculateReplyStatus /home/runner/work/xochitl/xochitl/src/xofm/libs/http/src/qt-transaction.cpp:162)
May 20 20:50:26 imx8mm-ferrari xochitl[10890]: 20:50:25.322 rm.network.transaction Request failed: RequestId=10730 error='Unspecified network error' http=none url=https://ping.remarkable.com/analytics/v2/events (finish /home/runner/work/xochitl/xochitl/src/xofm/libs/http/src/transaction.cpp:307)
May 20 20:50:26 imx8mm-ferrari xochitl[10890]: 20:50:25.322 rm.network.transaction RequestId=10730: transaction ended in 32990ms after following events:
May 20 20:50:26 imx8mm-ferrari xochitl[10890]: Create 2025-05-20 20:49:52.330638870
May 20 20:50:26 imx8mm-ferrari xochitl[10890]: Send 2025-05-20 20:49:52.330805868
May 20 20:50:26 imx8mm-ferrari xochitl[10890]: Fail 2025-05-20 20:50:25.321605039 (~Transaction /home/runner/work/xochitl/xochitl/src/xofm/libs/http/src/transaction.cpp:113)
When it's triggering, it ends up with about 100 RequestIds over a single second. I don't really see anything else around in the logs that's meaningful, and it just seems like it gets stuck in a bad networking state? I seem to recall that I saw a series of messages like this a couple of months ago and the solution was to just reinstall the proxy.
Reinstalling it makes the messages above go away, but it still ends up with 100% usage. I just see:
May 20 21:15:18 imx8mm-ferrari xochitl[17655]: 21:15:18.082 rm.network.notifications Notifications socket is not OK: UnconnectedState (checkIfShouldConnect /home/runner/work/xochitl/xochitl/src/notifications/src/notifications.cpp:284)
May 20 21:15:20 imx8mm-ferrari rm-sync[17656]: rm.synchronizer: -- execute requested: 2025-05-20T21:15:20.872
May 20 21:15:20 imx8mm-ferrari rm-sync[17656]: rm.synchronizer: -- running sync, batch: 0/1, minimal:true
May 20 21:15:21 imx8mm-ferrari rm-sync[17656]: rm.synchronizer: -- execute ended: 2025-05-20T21:15:21.205
I'm seeing this again, with similar logs to @Esras:
Jun 04 19:26:17 reMarkable rm-sync[292]: rm.synchronizer: -- execute requested: 2025-06-04T19:26:17.659
Jun 04 19:26:17 reMarkable rm-sync[292]: rm.synchronizer: -- running sync, batch: 0/1, minimal:true
Jun 04 19:26:18 reMarkable rm-sync[292]: rm.synchronizer: -- execute ended: 2025-06-04T19:26:18.427
Jun 04 19:26:21 reMarkable xochitl[243]: 19:26:21.867 qt.network.http2 stream 2001 finished with error: "Server stopped accepting new streams before this stream was established"
Jun 04 19:26:21 reMarkable xochitl[243]: 19:26:21.869 rm.network.qt RequestId=3559: Received QNetworkReply error code 205 (calculateReplyStatus /home/runner/work/xochitl/xochitl/src/xofm/libs/http/src/qt-transaction.cpp:162)
Jun 04 19:26:21 reMarkable xochitl[243]: 19:26:21.870 rm.network.transaction Request failed: RequestId=3559 error='Unspecified network error' http=none url=https://ping.remarkable.com/analytics/v2/events (finish /home/runner/work/xochitl/xochitl/src/xofm/libs/http/src/transaction.cpp:307)
Jun 04 19:26:21 reMarkable xochitl[243]: 19:26:21.870 rm.network.transaction RequestId=3559: transaction ended in 1681ms after following events:
Jun 04 19:26:21 reMarkable xochitl[243]: Create 2025-06-04 19:26:20.188465683
Jun 04 19:26:21 reMarkable xochitl[243]: Send 2025-06-04 19:26:20.188653301
Jun 04 19:26:21 reMarkable xochitl[243]: Fail 2025-06-04 19:26:21.869510693 (~Transaction /home/runner/work/xochitl/xochitl/src/xofm/libs/http/src/transaction.cpp:113)
Jun 04 19:26:21 reMarkable xochitl[243]: 19:26:21.870 rm.network.qt RequestId=3558: Received QNetworkReply error code 205 (calculateReplyStatus /home/runner/work/xochitl/xochitl/src/xofm/libs/http/src/qt-transaction.cpp:162)
Jun 04 19:26:21 reMarkable xochitl[243]: 19:26:21.871 rm.network.transaction Request failed: RequestId=3558 error='Unspecified network error' http=none url=https://ping.remarkable.com/analytics/v2/events (finish /home/runner/work/xochitl/xochitl/src/xofm/libs/http/src/transaction.cpp:307)
Jun 04 19:26:21 reMarkable xochitl[243]: 19:26:21.871 rm.network.transaction RequestId=3558: transaction ended in 1683ms after following events:
Jun 04 19:26:21 reMarkable xochitl[243]: Create 2025-06-04 19:26:20.187537966
Jun 04 19:26:21 reMarkable xochitl[243]: Send 2025-06-04 19:26:20.187724834
Jun 04 19:26:21 reMarkable xochitl[243]: Fail 2025-06-04 19:26:21.870574655 (~Transaction /home/runner/work/xochitl/xochitl/src/xofm/libs/http/src/transaction.cpp:113)
Jun 04 19:26:21 reMarkable xochitl[243]: 19:26:21.872 qt.network.http2 stream 2003 finished with error: "Server stopped accepting new streams before this stream was established"
Jun 04 19:26:21 reMarkable xochitl[243]: 19:26:21.872 qt.network.http2 stream 2005 finished with error: "Server stopped accepting new streams before this stream was established"
Jun 04 19:26:21 reMarkable xochitl[243]: 19:26:21.872 qt.network.http2 stream 2007 finished with error: "Server stopped accepting new streams before this stream was established"
...
Jun 04 19:26:21 reMarkable xochitl[243]: 19:26:21.886 rm.network.qt RequestId=3557: Received QNetworkReply error code 205 (calculateReplyStatus /home/runner/work/xochitl/xochitl/src/xofm/libs/http/src/qt-transaction.cpp:162)
Jun 04 19:26:21 reMarkable xochitl[243]: 19:26:21.887 rm.network.transaction Request failed: RequestId=3557 error='Unspecified network error' http=none url=https://ping.remarkable.com/analytics/v2/events (finish /home/runner/work/xochitl/xochitl/src/xofm/libs/http/src/transaction.cpp:307)
Jun 04 19:26:21 reMarkable xochitl[243]: 19:26:21.887 rm.network.transaction RequestId=3557: transaction ended in 1699ms after following events:
Jun 04 19:26:21 reMarkable xochitl[243]: Create 2025-06-04 19:26:20.186565375
Jun 04 19:26:21 reMarkable xochitl[243]: Send 2025-06-04 19:26:20.186783867
Jun 04 19:26:21 reMarkable xochitl[243]: Fail 2025-06-04 19:26:21.886488718 (~Transaction /home/runner/work/xochitl/xochitl/src/xofm/libs/http/src/transaction.cpp:113)
Jun 04 19:26:21 reMarkable xochitl[243]: 19:26:21.887 rm.network.qt RequestId=3556: Received QNetworkReply error code 205 (calculateReplyStatus /home/runner/work/xochitl/xochitl/src/xofm/libs/http/src/qt-transaction.cpp:162)
Jun 04 19:26:21 reMarkable xochitl[243]: 19:26:21.887 rm.network.transaction Request failed: RequestId=3556 error='Unspecified network error' http=none url=https://ping.remarkable.com/analytics/v2/events (finish /home/runner/work/xochitl/xochitl/src/xofm/libs/http/src/transaction.cpp:307)
Jun 04 19:26:21 reMarkable xochitl[243]: 19:26:21.888 rm.network.transaction RequestId=3556: transaction ended in 1704ms after following events:
Jun 04 19:26:21 reMarkable xochitl[243]: Create 2025-06-04 19:26:20.183010251
Jun 04 19:26:21 reMarkable xochitl[243]: Send 2025-06-04 19:26:20.183201994
Jun 04 19:26:21 reMarkable xochitl[243]: Fail 2025-06-04 19:26:21.887242817 (~Transaction /home/runner/work/xochitl/xochitl/src/xofm/libs/http/src/transaction.cpp:113)
Jun 04 19:26:21 reMarkable xochitl[243]: 19:26:21.888 rm.network.qt RequestId=3555: Received QNetworkReply error code 205 (calculateReplyStatus /home/runner/work/xochitl/xochitl/src/xofm/libs/http/src/qt-transaction.cpp:162)
Jun 04 19:26:21 reMarkable xochitl[243]: 19:26:21.888 rm.network.transaction Request failed: RequestId=3555 error='Unspecified network error' http=none url=https://ping.remarkable.com/analytics/v2/events (finish /home/runner/work/xochitl/xochitl/src/xofm/libs/http/src/transaction.cpp:307)
Jun 04 19:26:21 reMarkable xochitl[243]: 19:26:21.888 rm.network.transaction RequestId=3555: transaction ended in 1707ms after following events:
Jun 04 19:26:21 reMarkable xochitl[243]: Create 2025-06-04 19:26:20.180847577
Jun 04 19:26:21 reMarkable xochitl[243]: Send 2025-06-04 19:26:20.181031195
Jun 04 19:26:21 reMarkable xochitl[243]: Fail 2025-06-04 19:26:21.887898294 (~Transaction /home/runner/work/xochitl/xochitl/src/xofm/libs/http/src/transaction.cpp:113)
...
No document has failure logs this time around.
I'm not running the proxy, I've installed a root cert and use /etc/hosts override on the tablet.
Anything logged on the sever side then? Seems it has to do something with http/2, but no idea what. @rmitchellscott how is you kubernetes ingres setup?
I'm using ingress-nginx with the following helm values:
rbac:
create: true
controller:
allowSnippetAnnotations: true
enableAnnotationValidations: false
config:
annotations-risk-level: Critical
allow-cross-namespace-resources: true
strict-validate-path-type: false
proxy-buffer-size: 16k
use-gzip: "true"
hsts-max-age: "31536000"
hsts-preload: "true"
disable-ipv6: "true"
disable-ipv6-dns: "true"
addHeaders:
Referrer-Policy: same-origin, strict-origin-when-cross-origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
ingressClassResource:
default: true
externalTrafficPolicy: Local
admissionWebhooks:
enabled: false
As for the ingress itself:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
meta.helm.sh/release-name: remarkable-cloud
meta.helm.sh/release-namespace: remarkable-cloud
nginx.ingress.kubernetes.io/client-body-buffer-size: 128k
nginx.ingress.kubernetes.io/proxy-body-size: 100m
nginx.ingress.kubernetes.io/proxy-read-timeout: "10800"
name: remarkable-cloud
namespace: remarkable-cloud
spec:
ingressClassName: nginx
rules:
- host: remarkable.mydomain.tld
http:
paths:
- backend:
service:
name: remarkable-cloud
port:
number: 3000
path: /
pathType: Prefix
tls:
- hosts:
- remarkable.mydomain.tld
secretName: mydomain-wildcard
I have a second ingress, with the same annotations as the first, but all the additional hosts:
- hosts:
- hwr-production-dot-remarkable-production.appspot.com
- service-manager-production-dot-remarkable-production.appspot.com
- local.appspot.com
- my.remarkable.com
- internal.cloud.remarkable.com
- ping.remarkable.com
- backtrace-proxy.cloud.remarkable.engineering
In my case, the logs above "include" the proxy log, in that nothing is getting logged (I do not have any debug logging turned on with it, though). Adding some more info here that might not be helpful, but figured documenting it was good.
My webserver setup is an nginx doing a stream proxy_pass to another nginx instance that then proxy_passes to the actual container. The nginx config should be the default that ships with Fedora. The configs:
stream {
server {
listen {80|443};
proxy_pass {other_server}:{80|443};
}
...
}
The other nginx instance that has certs and actually sends it to the appropriate endpoint:
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 4096;
...
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
proxy_set_header X-Forwarded-Ssl $proxy_x_forwarded_ssl;
proxy_set_header X-Forwarded-Port $proxy_x_forwarded_port;
# Mitigate httpoxy attack (see README for details)
proxy_set_header Proxy "";
and that forwards it to the actual container running rmfakeproxy.
Some potentially irrelevant logs (on device wake, but not seeing the slowdown at this exact moment):
On the server side, I see a bunch (234 in one second) of POSTs to \"/analytics/v2/events\"", then:
time="2025-06-11T17:29:01Z" level=info msg="[auth-middleware] UserId: <username> deviceId: <deviceID> newSync: true"
time="2025-06-11T17:29:01Z" level=debug msg="Fullpath: /data/users/<username>/sync/root"
time="2025-06-11T17:29:01Z" level=info msg="[GIN] 2025/06/11 - 17:29:01 | 200 | 529.708µs | 10.89.1.14 | GET \"/sync/v4/root\""
then another 90 event posts in 1 second, then between 100 - 300 each second for the next about 8 seconds, then nothing.
Interestingly, this bug seems to have gone away with software version 3.19.0.82. I haven't seen it reoccur since then, but I will keep an eye on it.
I haven't seen this issue happen again, and I've now upgraded from3.18 to 3.20. I'm going to close this and will re-open or open a new issue if it happens again.
This is happening to me again, this time on 3.20.0.92 and the reMarkable Paper Pro. I'll attempt to gather some more logs. @Esras have you seen it again?
rmfakecloud version is now 0.0.25, I'm still set up using /etc/hosts overrides, no on-device proxy.
Mine is currently being RMAd, so I'm down a device to do testing on. When I get it back, I will test it again.
Here are logs I grabbed as I woke the device from sleep a bit ago.
ingress-nginx-20250730.log rmfakecloud-20250730.log rmpp-20250730.log
A couple things that jumped out at me:
Jul 30 16:41:28 imx8mm-ferrari xochitl[939]: 16:41:28.098 qt.network.http2 stream 2077 finished with error: "Server stopped accepting new streams before this stream was established"
Jul 30 16:41:28 imx8mm-ferrari xochitl[939]: 16:41:28.099 rm.http.transaction Request failed: RequestId=109700 error='Unspecified network error' http=none url=https://ping.remarkable.com/analytics/v2/events (finish /home/runner/work/xochitl/xochitl/src/xofm/libs/http/src/transaction.cpp:316)
I might have made some progress here. I did a capture from the official response for the /analytics/v2/events endpoint:
HTTP/2.0 201
rm-token-ttl-hint: 10701
content-type: text/plain; charset=utf-8
content-length: 22
date: Sat, 02 Aug 2025 05:16:39 GMT
server: Google Frontend
traceparent: 00-b0c2f9fc3023e433317f1f616b084f2b-f3257f529b7d6011-00
x-cloud-trace-context: b0c2f9fc3023e433317f1f616b084f2b/17520549918127513617
via: 1.1 google
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
{"message":"Success"}
I modified the following, and I haven't seen any logs about analytics failures since, and performance seems better:
diff --git a/internal/app/handlers.go b/internal/app/handlers.go
index 8c8575a..59daaaa 100644
--- a/internal/app/handlers.go
+++ b/internal/app/handlers.go
@@ -1181,7 +1181,7 @@ func (app *App) nullReport(c *gin.Context) {
// if err != nil {
// log.Warn("could not read report data")
// }
- c.Status(http.StatusOK)
+ c.JSON(http.StatusCreated, gin.H{"message": "Success"})
}
// / remove remarkable ads
Just changing from 200 to 201 didn't do the trick, but adding {"message": "Success"} had immediate effect for me.
I'll keep testing with this setup. If anyone else wants to give it a go, here's my test docker image with the change: git.scottlabs.io/scott-labs/rmfakecloud:events201-2
nullReport is used for the following endpoints, and I don't know if the others will take kindly to the different response:
/analytics/v2/events
/v1/reports
/v2/reports
/report/v1
/v2/events
Edit to add: changing my /etc/hosts line to 0.0.0.0 ping.remarkable.com might have provided an on-device workaround, I've got a single error line instead of hundreds:
Aug 02 06:19:34 imx8mm-ferrari xochitl[7297]: 06:19:34.669 rm.http.transaction Request failed: RequestId=5 error='Host not found error' http=none url=https://ping.remarkable.com/analytics/v2/events (finish /home/runner/work/xochitl/xochitl/src/xofm/libs/http/src/transaction.cpp:316)
Aug 02 06:19:34 imx8mm-ferrari xochitl[7297]: 06:19:34.669 rm.http.transaction RequestId=5: transaction ended in 13ms after following events:
Aug 02 06:19:34 imx8mm-ferrari xochitl[7297]: Create 2025-08-02 06:19:34.655675750
Aug 02 06:19:34 imx8mm-ferrari xochitl[7297]: Send 2025-08-02 06:19:34.655962125
Aug 02 06:19:34 imx8mm-ferrari xochitl[7297]: Fail 2025-08-02 06:19:34.668732875 (~Transaction /home/runner/work/xochitl/xochitl/src/xofm/libs/http/src/transaction.cpp:113)
I'll add this change for the route /analytics/v2/events. Until now it didn't seem to matter, but if it helped, we need to mimic the response of those routes too.
As of right now, I'm not seeing this issue on 3.20.0.92, running the on-device proxy. I'll keep checking and an eye out. I do still see:
Aug 10 19:40:54 imx8mm-ferrari xochitl[22745]: 19:40:54.053 rm.network.notifications Connected to notifications server (onStateChanged /home/runner/work/xochitl/xochitl/src/notifications/src/notifications.cpp:206)
Aug 10 19:40:58 imx8mm-ferrari xochitl[22745]: 19:40:58.603 rm.network.notifications Notifications socket is not OK: UnconnectedState (checkIfShouldConnect /home/runner/work/xochitl/xochitl/src/notifications/src/notifications.cpp:260)
Aug 10 19:40:58 imx8mm-ferrari xochitl[22745]: 19:40:58.603 rm.network.notifications Notifications socket is not OK: UnconnectedState (checkIfShouldConnect /home/runner/work/xochitl/xochitl/src/notifications/src/notifications.cpp:260)
in the logs, but significantly less than the logs posted above (also not the same message).
I have seen no reoccurrence of performance issues since moving to my image with the fix over a week ago, nor any Unspecified network error logs.