dealer icon indicating copy to clipboard operation
dealer copied to clipboard

timeout: increase latency

Open nicolasburtey opened this issue 2 years ago • 2 comments

there is a lot of timeout and related alert. increasing latency (time for the client to get a response) may help.

cc @ntheile

nicolasburtey avatar Jul 27 '22 14:07 nicolasburtey

@sebastienverreault

Here is a link to the HTTP trace that takes about 28 seconds to complete. https://ui.honeycomb.io/galoy/datasets/galoy-staging/result/HQiS4jQr8Vu/trace/DJT7RQJDRrC?fields[]=c_name&fields[]=c_service.name&span=bedb8bee50d4762b . This appears to be alerting about 10 times each hour.

It looks like the HTTP get call to https://www.okx.com/api/v5/account/balance took about 10 seconds during this time.

ntheile avatar Jul 27 '22 14:07 ntheile

Here is another example of a timeout for the same HTTP GET call (10 second timeout) https://www.okx.com/api/v5/account/balance from this trace service.name:galoy-bbw-dealer,name:app.exchangeBase.fetchBalance,error.name:RequestTimeout....but the trace originated from a few different places, causing three separate alerts in pager duty:

  1. name:app.exchangeBase.fetchBalance,error.name:RequestTimeout
  2. name:app.okexExchange.closePosition,error.name:RequestTimeout
  3. name:app.exchangeBase.createMarketOrder

https://ui.honeycomb.io/galoy/datasets/galoy-bbw/result/h9BbZxnEnDN/trace/f7dErqKHSY5?fields[]=c_name&fields[]=c_service.name&span=e05be23255ea7752

ntheile avatar Jul 28 '22 00:07 ntheile