NEPs
NEPs copied to clipboard
near
[Discussion] Shard congestion issue
The concern is to have a shard congested with delayed receipt and the gas price doesn't react fast enough to slow down the congestion. It's possible to spam the block with 500 TX which all going to target one expensive contract and slow down this shard for 100 blocks. The cost of the attack will increase over time because the price of gas will be adjusted, but after X number of blocks the prepaid gas cost will not be higher than the actual current price, which creates the deficit. This makes the attack less expensive than it should be.
This was partially addressed by changing the way we calculate the gas price at the receipt is created. The gas price difference is refunded back once the receipt is executed. Currently, we use the pessimistic inflation of 1.03 per block up to 64 blocks with the estimate of at most 5TGas per block. But if there are delayed receipts in the shard, then the entire execution might be delayed for undefined amount of blocks, so the purchase price of gas during the transaction conversion to receipt will not be enough.
There are some ideas on how to address this issue:
Burn all attached gas.
This way we can fill the blocks properly to never exceed the overall capacity of the processing power. This solution has a few issues:
- The DevX becomes a pain. You need to precisely estimate gas for cross-contract calls to avoid overcharging the user, but also if you underestimate, the chain of calls might fail at unexpected state.
- It doesn't fully solve the shard congesting, because multiple shards might route receipts to one shard and still create a delayed queue. But this delayed queue can only be blocked by N blocks (where N is the number of shards) per block.
TX/Receipt Priority based on gas price
We can introduce receipt and transaction ordering based on the gas price, so a receipt with higher gas price will be processed first. It also has a few issues:
- We'll introduce explicit gas price in a transaction. So people will need to think about which gas price to put.
- Every receipt will also be ordered based on the gas price, so some receipts can be delayed for infinite amount of time. This can be addressed through a price boosting, where a receipt boosts the gas price of another receipt.
- Receipts will not come in order. This is already somewhat an issue with delayed queue that messes the order of local receipts.
TX can define max_gas_price and validators can decide whether to include such transaction to a chunk
(Not a solid idea)
It's similar to current approach, except instead of 1.03 inflation, you define the max price at which the TX is ready to be included and then buy gas at that price. Validators decide whether to include this transaction into a chunk, because it can cause the deficit later on, if the max_gas_price is too low and there are a delayed shard.
@mikedotexe @chadoh @amgando @potatodepaulo @kcole16 @vgrichina This will mess up our DevX independently on the solution so either way you should be aware of it. The least damaging proposal "TX/Receipt Priority based on gas price" will mean that some of our receipts might end up forever in the queue, so we should allow people to bump them by dropping some tokens on them, which would require Applayer support
@evgenykuzyakov if we have a priority we should decide on how to handle the ones with lower priority. If we still store them in the state we can potentially be attacked by someone generating a lot of low priority receipts.
Our DevX is already in pretty sad shape for everything involving cross-contract calls. :/
How about instead of discussing the abstract issues we work out through some real life examples in say Defi.
E.g. what does it take to build a DEX on NEAR and how each of the proposed approaches affects it. Or even better work through some more complex example where somebody wants to use multiple automated market makers together. Taking into accounts rollbacks, front-running, etc.
It's possible to spam the block with 500 TX which all going to target one expensive contract and slow down this shard for 100 blocks.
Not sure why targeting one expensive contract is important part here. What is different vs targeting 500 different contracts?
if we have a priority we should decide on how to handle the ones with lower priority. If we still store them in the state we can potentially be attacked by someone generating a lot of low priority receipts.
It can be resolved by holding receipts on the original shard, if the destination shard is congested. Every shard can report them minimum gas price at which the next delayed receipt has to be executed, so the other shards can't forward receipts with gas price lower than this price.
The problem only exists if it's possible to keep spamming the network with low cost transactions targeting the same shard. It seems we still need some combination of minimum gas_price to avoid taking new transactions with low gas price and priority based on gas_price to be able to execute more important receipts on the given shard.
Re-sharding should address the problem of low-priority delayed receipts by splitting them into multiple shards and increasing throughput.
DEX examples
DEX implementation that target current runtime (without safes) will not be affected much in either scenario. Since there are no locks and not much cross-contract calls required for trades. You pretty much have to transfer asset to a DEX first before you can trade it. Of course you can try to do this within one receipt through allowances, but the asset you want to buy has to be owned by the DEX itself.
-
Burn all attached gas -> doesn't affect at all, but makes it miserable for a developer. It requires to precisely estimate fees and sometimes it might not be possible (e.g. voting contract).
-
Priority tx/receipts based on gas price -> without locks, it doesn't introduce attacks unless you make a market order. If you do a market order, then you define maximum slippage and some front-runner can try to sell it to you at maximum slippage (so it's like a limit order). As for receipt being stuck for a while, it doesn't affect the exchange and usually solved by order expiration. E.g. you taker trade can only be executed within 20 seconds of X, otherwise it's noop. So if someone front-runs you and blocks it, they can only do this for 20 seconds.
-
max_gas_price -> It doesn't fully address the issue.
Not sure why targeting one expensive contract is important part here. What is different vs targeting 500 different contracts?
That's just the way of saying multiple shards can attack one shard. It's similar to target 500 different account on the same shard.
Discussed priority receipts with @nearmax . Want to clarify some things and propose a few new things.
- Each transaction should specify the
min_gas_priceandmax_gas_price. Thegas_priceat which the transaction is converted to a receipt will be determined based onmin_gas_priceof the shard. - Each receipt will have an explicit
gas_priceat which it was created. It'll burn gas at this price instead of the current block gas price. There will be no more current blockgas_price. - Each shard will have
min_gas_price. It's different from current block-levelgas_pricein two ways.-
min_gas_pricedefines the minimum gas price on the given shard to convert a transaction or receive an incoming receipt. To convert a transaction to a receipt on this shard, the transaction gas price range should include currentmin_gas_priceof the shard. In order to forward a receipt to the destinationshard, the receipt has to have thegas_pricenot smaller than the lastmin_gas_price. -
min_gas_pricewill be determined based on the congesting of the shard. It's not going to be increased over time, but will be determined based on the number of delayed receipts (both incoming and outgoing).
-
NOTE: We need to figure out a way of blocking outgoing receipts because of the circular dependencies
To avoid circular dependencies we should let shards forward outgoing receipts at any price if the incoming queue of delayed receipts on the destination shard is empty.
Each shard will have to publish the following info to the block:
- minimum gas price for new transactions
- minimum gas price for incoming receipts.
We should keep the global min_gas_price similar to the one we have right now to avoid being spammed with cheap transactions.
DEX implementation that target current runtime (without safes) will not be affected much in either scenario. Since there are no locks and not much cross-contract calls required for trades.
let's find some real life example which actually would rely on cross-contract calls?
let's find some real life example which actually would rely on cross-contract calls?
Deposit to DEX requires cross-contract call to token contract. It can be delayed if either DEX or the token contract is shards are congested.
min_gas_price will be determined based on the congesting of the shard. It's not going to be increased over time, but will be determined based on the number of delayed receipts (both incoming and outgoing).
Some questions:
- How exactly is this determined?
- Since it seems that this price cannot change faster than one block, an attacker can adaptively adjust their price in transaction to congest the shard the delayed receipts.
- Are those delayed outgoing receipts stored in state as well? It seems that we need some sort of btree map with duplicate keys.
How exactly is this determined?
I think transaction TPS can be determined on the size of the outgoing queue(s). The more receipts in the queue the more transaction cost should be. Also it can be determined based on the minimum gas price of receiving shards. E.g. if the outgoing queue is blocked because of the gas pricing, then it should influence the minimum transaction gas price.
Since it seems that this price cannot change faster than one block, an attacker can adaptively adjust their price in transaction to congest the shard the delayed receipts.
It's not inflationary anymore. So if the shard is congested, the transaction costs can go up in two blocks to prevent congestion.
Are those delayed outgoing receipts stored in state as well? It seems that we need some sort of btree map with duplicate keys.
Yep, we'll need to order them based on the gas_price.
After giving it some thought – from DevX POV it might be reasonable if child transactions can fail because gas cost increased. Motivation: as a developer you should anyway be ready for them to fail (for any other reason) and roll back.
After giving it some thought – from DevX POV it might be reasonable if child transactions can fail because gas cost increased. Motivation: as a developer you should anyway be ready for them to fail (for any other reason) and roll back.
But it also means your own callback may fail because the gas increased while you called some other shard. It will make it impossible to do anything including rollbacks.
But it also means your own callback may fail
Yes, so maybe we actually have to embrace it and make it easy to unscrew the situation vs try to make sure it doesn't happen. As it seems like it still can happen, we are just making it more of an edge case.
It will make it impossible to do anything including rollbacks.
I assume you mean without sending new transactions, correct?
In any case that is more of thought experiment for now. But let's think through the case where we allow everything to fail and then let client recover. Maybe whole system still is simpler / more robust.
No. The issue is if you can't rely on your own callback to be executed, you can't do anything in async system. It's like Eth pre-rollback where you can terminate contract execution and the necessary spot and keep the state for the first part, but not for the last.
So it's better that you can rely on the receipt being stuck in the low priority queue than failing. Even a simple operation like transferFrom may just lose tokens if the callback is not 100% handled.
Current async model works on the assumption that the receipts are never lost, so it's eventually consistent.
So it's better that you can rely on the receipt being stuck in the low priority queue than failing.
Depending on how long it can be stuck it's not necessarily better. If it can be stuck for days, I would prefer failure with an option to get it unstuck.