dealer
dealer copied to clipboard
timeout: increase latency
there is a lot of timeout and related alert. increasing latency (time for the client to get a response) may help.
cc @ntheile
@sebastienverreault
Here is a link to the HTTP trace that takes about 28 seconds to complete. https://ui.honeycomb.io/galoy/datasets/galoy-staging/result/HQiS4jQr8Vu/trace/DJT7RQJDRrC?fields[]=c_name&fields[]=c_service.name&span=bedb8bee50d4762b . This appears to be alerting about 10 times each hour.
It looks like the HTTP get call to https://www.okx.com/api/v5/account/balance
took about 10 seconds during this time.
Here is another example of a timeout for the same HTTP GET call (10 second timeout) https://www.okx.com/api/v5/account/balance
from this trace service.name:galoy-bbw-dealer,name:app.exchangeBase.fetchBalance,error.name:RequestTimeout
....but the trace originated from a few different places, causing three separate alerts in pager duty:
-
name:app.exchangeBase.fetchBalance,error.name:RequestTimeout
-
name:app.okexExchange.closePosition,error.name:RequestTimeout
-
name:app.exchangeBase.createMarketOrder
https://ui.honeycomb.io/galoy/datasets/galoy-bbw/result/h9BbZxnEnDN/trace/f7dErqKHSY5?fields[]=c_name&fields[]=c_service.name&span=e05be23255ea7752