dx-services `replacement transaction underpriced` for `claimFunds`

It happened to me a few minutes ago that claiming funds failed with an replacement transaction underpriced error. The next run of claimFunds was successful.

  INFO-dx-service:tasks:claimFunds Claiming funds for pre +0ms
Secp256k1 bindings are not compiled. Pure JS implementation will be used.
  INFO-dx-service:tasks:claimFunds Claiming for address 0xc32cac63823b556e6ebf61bb74149f08bf1aab34 affected bots: Main buyer bot, Main seller bot +3s
  INFO-dx-service:services:DxTradeService I have to claim from 1 auctions as seller +0ms
  INFO-dx-service:services:DxTradeService Claiming as seller MKR-DAI for auctions: [ BigNumber { s: 1, e: 0, c: [ 3 ] } ] +1ms
Error: replacement transaction underpriced
    at /usr/src/app/node_modules/web3-provider-engine/subproviders/provider.js:19:36
    at XMLHttpRequest.request.onreadystatechange (/usr/src/app/node_modules/web3/lib/web3/httpprovider.js:118:13)
    at XMLHttpRequestEventTarget.dispatchEvent (/usr/src/app/node_modules/xhr2/lib/xhr2.js:64:18)
    at XMLHttpRequest._setReadyState (/usr/src/app/node_modules/xhr2/lib/xhr2.js:354:12)
    at XMLHttpRequest._onHttpResponseEnd (/usr/src/app/node_modules/xhr2/lib/xhr2.js:509:12)
    at IncomingMessage.<anonymous> (/usr/src/app/node_modules/xhr2/lib/xhr2.js:469:24)
    at emitNone (events.js:111:20)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)
Error: replacement transaction underpriced
    at /usr/src/app/node_modules/web3-provider-engine/subproviders/provider.js:19:36
    at XMLHttpRequest.request.onreadystatechange (/usr/src/app/node_modules/web3/lib/web3/httpprovider.js:118:13)
    at XMLHttpRequestEventTarget.dispatchEvent (/usr/src/app/node_modules/xhr2/lib/xhr2.js:64:18)
    at XMLHttpRequest._setReadyState (/usr/src/app/node_modules/xhr2/lib/xhr2.js:354:12)
    at XMLHttpRequest._onHttpResponseEnd (/usr/src/app/node_modules/xhr2/lib/xhr2.js:509:12)
    at IncomingMessage.<anonymous> (/usr/src/app/node_modules/xhr2/lib/xhr2.js:469:24)
    at emitNone (events.js:111:20)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)

Any idea what may cause it...? Is it because the sell/buy bots are running simultaneously using the same address - some nonce conflict maybe...?

Aug 29 '18 21:08 ghost

It's just that. It sends two transactions too close. The "default" nonce used by the TruffleHDWallet is "pending", meaning that if it's in the pool, it'll get the correct nonce.

We experience this, from time to time, and I think is because infura has a load balancer, and we send the transactions to close.

We could try to space them a bit in time, since time is not so critical for this operations. I guess with some seconds we give time to the nodes to catch up.

@dasanra , can you take a look into this?

Aug 30 '18 09:08 anxolin

Are you raising gas price if transaction hasn't confirmed for some time? Or is it about two different transactions? Maybe if it's really about replacement transaction, Infura requires some minimum increment (I think Parity requires the new gas price to be at least +10% compared to the old one)?

Aug 30 '18 12:08 ghost

@reverendus I have some doubts regarding this issue. We were currently working on a new feature of automatic retry if gas price was too low but we had some issues with it and we had to disable it in one of the latest versions. So could you confirm that you are using the latest staging version of the bots?

To check this the bots have and endpoint that gives you info about the running instance. You can check it in http://hostnameorip:8081/api/about

In version it should be using 0.6.8 If not the case you should stop the instance and run it with the same command. It should check that it's not using the latest docker image and then it should show in the logs the download of the latest one

Aug 30 '18 15:08 dasanra

It hasn't happened to me again since I originally reported it, I am sure I'm running claimFunds from the 0.6.8 docker image at the moment, not completely sure what version was it previously when this problem happened.

I will wait a few more days, if it doesn't happen again I'll close the issue.

Aug 30 '18 21:08 ghost

It just happened to me again:

2018-09-03T05:32:41.718Z INFO-dx-service:bots:BuyLiquidityBot-MKR-DAI I've bought 287.83386722976017449045 DAI (512.4630399992003 USD) in auction 20 to ensure BUY liquidity
2018-09-03T05:32:43.916Z WARN-dx-service:bots:AuctionEventWatcher-0x2d9585690a724bfa29a212295a9e8c714ca694aa-0x62f25065ba60ca3a2044344955a3b2530e355111 One auction cleared, but it was for a known pair: 0x2d9585690a724bfa29a212295a9e8c714ca694aa-0x62f25065ba60ca3a2044344955a3b2530e355111
2018-09-03T05:33:19.222Z INFO-dx-service:services:LiquidityService-MKR-DAI Not enough liquidity for auction 21: MKR=$0, DAI=$0
2018-09-03T05:33:19.800Z INFO-dx-service:services:LiquidityService-MKR-DAI Selling 1.2162922304751524 MKR ($1005.03)
2018-09-03T05:33:21.471Z ERROR-dx-service:repositories:AuctionRepoImpl Error on transaction "postSellOrder", from "0xc32cac63823b556e6ebf61bb74149f08bf1aab34". Params: [0x2D9585690A724BfA29a212295a9E8C714ca694aA,0x62f25065BA60CA3A2044344955A3B2530e355111,21,1216292230475152346]. Gas: 305668, GasPrice: 1000000000. Error: Error: replacement transaction underpriced
Error: replacement transaction underpriced
    at /usr/src/app/node_modules/web3-provider-engine/subproviders/provider.js:19:36
    at XMLHttpRequest.request.onreadystatechange (/usr/src/app/node_modules/web3/lib/web3/httpprovider.js:118:13)
    at XMLHttpRequestEventTarget.dispatchEvent (/usr/src/app/node_modules/xhr2/lib/xhr2.js:64:18)
    at XMLHttpRequest._setReadyState (/usr/src/app/node_modules/xhr2/lib/xhr2.js:354:12)
    at XMLHttpRequest._onHttpResponseEnd (/usr/src/app/node_modules/xhr2/lib/xhr2.js:509:12)
    at IncomingMessage.<anonymous> (/usr/src/app/node_modules/xhr2/lib/xhr2.js:469:24)
    at emitNone (events.js:111:20)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)
2018-09-03T05:33:21.471Z ERROR-dx-service:bots:SellLiquidityBot-MKR-DAI There was an error ensuring sell liquidity with the account 0xc32cac63823b556e6ebf61bb74149f08bf1aab34: Error: replacement transaction underpriced
Error: replacement transaction underpriced
    at /usr/src/app/node_modules/web3-provider-engine/subproviders/provider.js:19:36
    at XMLHttpRequest.request.onreadystatechange (/usr/src/app/node_modules/web3/lib/web3/httpprovider.js:118:13)
    at XMLHttpRequestEventTarget.dispatchEvent (/usr/src/app/node_modules/xhr2/lib/xhr2.js:64:18)
    at XMLHttpRequest._setReadyState (/usr/src/app/node_modules/xhr2/lib/xhr2.js:354:12)
    at XMLHttpRequest._onHttpResponseEnd (/usr/src/app/node_modules/xhr2/lib/xhr2.js:509:12)
    at IncomingMessage.<anonymous> (/usr/src/app/node_modules/xhr2/lib/xhr2.js:469:24)
    at emitNone (events.js:111:20)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)
2018-09-03T05:33:21.472Z ERROR-dx-service:error Uncought promise rejection: Error: replacement transaction underpriced
Error: replacement transaction underpriced
    at /usr/src/app/node_modules/web3-provider-engine/subproviders/provider.js:19:36
    at XMLHttpRequest.request.onreadystatechange (/usr/src/app/node_modules/web3/lib/web3/httpprovider.js:118:13)
    at XMLHttpRequestEventTarget.dispatchEvent (/usr/src/app/node_modules/xhr2/lib/xhr2.js:64:18)
    at XMLHttpRequest._setReadyState (/usr/src/app/node_modules/xhr2/lib/xhr2.js:354:12)
    at XMLHttpRequest._onHttpResponseEnd (/usr/src/app/node_modules/xhr2/lib/xhr2.js:509:12)
    at IncomingMessage.<anonymous> (/usr/src/app/node_modules/xhr2/lib/xhr2.js:469:24)
    at emitNone (events.js:111:20)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)
2018-09-03T05:34:19.527Z INFO-dx-service:services:LiquidityService-MKR-DAI Not enough liquidity for auction 21: MKR=$0, DAI=$0
2018-09-03T05:34:20.109Z INFO-dx-service:services:LiquidityService-MKR-DAI Selling 1.2162922304751524 MKR ($1005.03)
2018-09-03T05:34:25.933Z INFO-dx-service:services:LiquidityService-MKR-DAI Posted a sell order. Transaction: 0xa9a590cac137373fcc32b22d577a7d4a6e400811156d1d213689cfdef5d58149
2018-09-03T05:34:25.933Z INFO-dx-service:bots:SellLiquidityBot-MKR-DAI I've sold 1.216292230475152346 MKR (1005.03 USD) in auction 21 to ensure SELL liquidity
2018-09-03T05:34:25.934Z WARN-dx-service:bots:SellLiquidityBot-MKR-DAI The sell liquidity was enssured by the routine check. Make sure there's no problem getting events
2018-09-03T05:35:17.747Z WARN-dx-service:services:LiquidityService-MKR-DAI There is a concurrent sell check going on, so no aditional check should be done
2018-09-03T05:36:17.676Z WARN-dx-service:services:LiquidityService-MKR-DAI There is a concurrent sell check going on, so no aditional check should be done

Sep 03 '18 07:09 ghost

pretty sure it's 0.6.8 now

Sep 03 '18 07:09 ghost

Happened again:

2018-09-11T10:26:47.035Z INFO-dx-service:services:LiquidityService-MKR-WETH The auction has 51.41 % of the buy volume bought. So we are good
2018-09-11T10:27:46.998Z INFO-dx-service:services:LiquidityService-MKR-WETH We need to ensure that 50 % of the buy volume is bought. Market Price: 1.765, Price: 1.6945646228686018, Relation: 96 %
2018-09-11T10:27:46.999Z INFO-dx-service:services:LiquidityService-MKR-WETH The auction has 51.5 % of the buy volume bought. So we are good
2018-09-11T10:28:46.979Z INFO-dx-service:services:LiquidityService-MKR-WETH We need to ensure that 100 % of the buy volume is bought. Market Price: 1.765, Price: 1.6914294424050123, Relation: 95.83 %
2018-09-11T10:28:46.979Z INFO-dx-service:services:LiquidityService-MKR-WETH The auction has 51.6 % of the buy volume bought. So we need to buy an extra 48.4 %
2018-09-11T10:28:47.259Z INFO-dx-service:services:LiquidityService-MKR-WETH Posting a buy order for 0.9733045276985203 WETH ($486.6522638492601)
2018-09-11T10:28:59.109Z INFO-dx-service:services:LiquidityService-MKR-WETH Posted a buy order. Transaction: 0x62b6dbb31b3a34ee81e1f0e5731b4f52333b9a30aaad724828297fbb2c69341b
2018-09-11T10:28:59.110Z INFO-dx-service:bots:BuyLiquidityBot-MKR-WETH I've bought 0.97330452769852028744 WETH (486.6522638492601 USD) in auction 49 to ensure BUY liquidity
2018-09-11T10:29:00.803Z WARN-dx-service:bots:AuctionEventWatcher-0x2d9585690a724bfa29a212295a9e8c714ca694aa-0xc778417e063141139fce010982780140aa0cd5ab One auction cleared, but it was for a known pair: 0x2d9585690a724bfa29a212295a9e8c714ca694aa-0xc778417e063141139fce010982780140aa0cd5ab
2018-09-11T10:29:33.557Z INFO-dx-service:services:LiquidityService-MKR-WETH Not enough liquidity for auction 50: MKR=$0, WETH=$0
2018-09-11T10:29:34.192Z INFO-dx-service:services:LiquidityService-MKR-WETH Selling 1.1889297586859569 MKR ($1005.03)
2018-09-11T10:29:35.545Z ERROR-dx-service:repositories:AuctionRepoImpl Error on transaction "postSellOrder", from "0xc32cac63823b556e6ebf61bb74149f08bf1aab34". Params: [0x2D9585690A724BfA29a212295a9E8C714ca694aA,0xc778417e063141139fce010982780140aa0cd5ab,50,1188929758685956740]. Gas: 300436, GasPrice: 1000000000. Error: Error: replacement transaction underpriced
Error: replacement transaction underpriced
    at /usr/src/app/node_modules/web3-provider-engine/subproviders/provider.js:19:36
    at XMLHttpRequest.request.onreadystatechange (/usr/src/app/node_modules/web3/lib/web3/httpprovider.js:118:13)
    at XMLHttpRequestEventTarget.dispatchEvent (/usr/src/app/node_modules/xhr2/lib/xhr2.js:64:18)
    at XMLHttpRequest._setReadyState (/usr/src/app/node_modules/xhr2/lib/xhr2.js:354:12)
    at XMLHttpRequest._onHttpResponseEnd (/usr/src/app/node_modules/xhr2/lib/xhr2.js:509:12)
    at IncomingMessage.<anonymous> (/usr/src/app/node_modules/xhr2/lib/xhr2.js:469:24)
    at emitNone (events.js:111:20)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)
2018-09-11T10:29:35.546Z ERROR-dx-service:bots:SellLiquidityBot-MKR-WETH There was an error ensuring sell liquidity with the account 0xc32cac63823b556e6ebf61bb74149f08bf1aab34: Error: replacement transaction underpriced
Error: replacement transaction underpriced
    at /usr/src/app/node_modules/web3-provider-engine/subproviders/provider.js:19:36
    at XMLHttpRequest.request.onreadystatechange (/usr/src/app/node_modules/web3/lib/web3/httpprovider.js:118:13)
    at XMLHttpRequestEventTarget.dispatchEvent (/usr/src/app/node_modules/xhr2/lib/xhr2.js:64:18)
    at XMLHttpRequest._setReadyState (/usr/src/app/node_modules/xhr2/lib/xhr2.js:354:12)
    at XMLHttpRequest._onHttpResponseEnd (/usr/src/app/node_modules/xhr2/lib/xhr2.js:509:12)
    at IncomingMessage.<anonymous> (/usr/src/app/node_modules/xhr2/lib/xhr2.js:469:24)
    at emitNone (events.js:111:20)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)
2018-09-11T10:29:35.547Z ERROR-dx-service:error Uncought promise rejection: Error: replacement transaction underpriced
Error: replacement transaction underpriced
    at /usr/src/app/node_modules/web3-provider-engine/subproviders/provider.js:19:36
    at XMLHttpRequest.request.onreadystatechange (/usr/src/app/node_modules/web3/lib/web3/httpprovider.js:118:13)
    at XMLHttpRequestEventTarget.dispatchEvent (/usr/src/app/node_modules/xhr2/lib/xhr2.js:64:18)
    at XMLHttpRequest._setReadyState (/usr/src/app/node_modules/xhr2/lib/xhr2.js:354:12)
    at XMLHttpRequest._onHttpResponseEnd (/usr/src/app/node_modules/xhr2/lib/xhr2.js:509:12)
    at IncomingMessage.<anonymous> (/usr/src/app/node_modules/xhr2/lib/xhr2.js:469:24)
    at emitNone (events.js:111:20)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)

Sep 11 '18 10:09 ghost

@reverendus thank you for the feedback. We found an scenario that should be causing this error. It's not critical, as is shown when trying to buy twice. We check liquidity too fast and the information with the previous buy may not be in the queried ethereum node (so it tries to buy again when is not necessary). We are looking how to deal with this and a patch will be released in a future version. This issue will remain open and we wil notify here when the patch is delivered.

Sep 11 '18 10:09 dasanra

@reverendus I also read that this happened with claimFunds, is a similar case, we execute two transactions very close and so one is rejected because the information in the ethereum node is not updated yet.

Sep 11 '18 10:09 dasanra

thanks for the update. it's not critical to us by any means, I just felt obliged to report it.

Sep 11 '18 23:09 ghost

dx-services dx-services copied to clipboard

`replacement transaction underpriced` for `claimFunds`

dx-services
dx-services copied to clipboard