Usage not being updated

[someoneIG]

Hi My 3 servers have a problem They don't count the traffic usage anymore :/

How can I solve this? I didn't have this problem at first

https://pasteboard.co/KeVbQ2p.png

https://pasteboard.co/K4JrnOr.png

[fortuna]

It's hard to tell. Can you SSH into the machine? Perhaps the metrics service is crashing. docker logs shadowbox from inside the machine should give you a hint.

[someoneIG]

photo of docker logs shadowbox

https://pasteboard.co/Kguq4WE.png https://pasteboard.co/KgurJzC.png

[fortuna]

The error message is from here: https://github.com/Jigsaw-Code/outline-server/blob/583ef409cd405cbdd2bcae95f9585017fb48cc73/src/shadowbox/server/server_access_key.ts#L111

It seems we are getting HTTP error 422 from Prometheus. The prometheus documentation says "422 Unprocessable Entity - when an expression can't be executed (RFC4918).", which is not very helpful.

Perhaps the next thing to try is to query Prometheus. From the server, you can try:

curl 'http://localhost:9090/api/v1/query?query=up'

And see what you get.

[someoneIG]

I got the following output

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"up","instance":"127.0.0.1:9090","job":"prometheus"},"value":[1629377495.004,"1"]},{"metric":{"__name__":"up","instance":"127.0.0.1:9091","job":"outline-server-main"},"value":[1629377495.004,"1"]},{"metric":{"__name__":"up","instance":"127.0.0.1:9092","job":"outline-server-ss"},"value":[1629377495.004,"1"]}]}}```

[someoneIG]

This happens after several days of use

On a server I just installed, I did not see an error with docker logs shadowbox

And this is the output of curl 'http://localhost:9090/api/v1/query?query=up' in this new server

curl 'http://localhost:9090/api/v1/query?query=up'
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"up","instance":"127.0.0.1:9090","job":"prometheus"},"value":[1629378344.061,"1"]},{"metric":{"__name__":"up","instance":"127.0.0.1:9091","job":"outline-server-main"},"value":[1629378344.061,"1"]},{"metric":{"__name__":"up","instance":"127.0.0.1:9092","job":"outline-server-ss"},"value":[1629378344.061,"1"]}]}}

[krakazyabra]

confirm, that's still happening. after some time prom stops to scrape 9091/9092 endpoints. little inspection shows, there are no metrics on that paths. there were some interesting messages, but i'm not sure they're connected with this issue:

(node:12) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.

and

(node:12) [DEP0111] DeprecationWarning: Access to process.binding('http_parser') is deprecated.
(Use `node --trace-deprecation ...` to show where the warning was created)

prom itself starts fine, loading the wal. most of all, if I add custom target (f.e. node-exporter) to its config, it starts to use it and scrapes node-exporter.

So, in my opinion, there is something strange with metrics from SS and outline, why they cannot be exposed.

[Leecns]

+1

[Leecns]

@fortuna

[maddyhof]

closing this issue in favor of #1188