livepeer
Arbitrum Gas Spikes Cause PPP to Exceed Limits and Drop Sessions
Summary
When Arbitrum gas briefly spikes, the orchestrator raises its price per pixel (PPP) to match the instantaneous gas price. If PPP exceeds the broadcaster’s configured ceiling (e.g. 1200 wei), all active sessions are dropped, even though gas usually normalizes within seconds.
Root Cause
In pm/recipient.go, faceValue() and txCost() derive PPP directly from the current gas price. The hard-coded fallback avgGasPrice = 3 gwei, tuned for legacy Arbitrum, is inaccurate for Nitro, leading to an incorrect gas estimate. As a result, the calculated PPP can briefly appear to overshoot broadcaster limits and trigger job cancellations even though it should not. If orchestrators have autoAdjustPrice=true enabled it will amplify the issue.
https://github.com/livepeer/go-livepeer/blob/9cbad8221f965a6ad7c71bb7e731ed680f5a70af/pm/recipient.go#L311-L313
Impact
- Active streams are interrupted and must reconnect.
- Orchestrators lose revenue during the outage.
Recommended Fix
-
Adjust the fallback: Raise the hard-coded
avgGasPriceto a level appropriate for Arbitrum Nitro. -
Make it dynamic: Replace the static fallback with a rolling average (e.g., from
eth_feeHistory) to smooth transient spikes (see https://github.com/livepeer/go-livepeer/issues/2078). -
Use
maxGasPriceto avoid drops: Update the software so that when gas exceeds the configuredmaxGasPrice, the orchestrator continues transcoding at the fixed pixel price and redeems tickets later once gas falls, preventing session loss (see https://github.com/livepeer/go-livepeer/issues/1966#issuecomment-915561648). -
Optional flag to relax sender-reserve checks: Add a startup flag such as
--ignoreSenderReserveRequirementsso orchestrators can choose to verify only that the broadcaster’s deposit covers the ticket face value, ignoring full sender-reserve requirements.
Related to
- https://github.com/livepeer/go-livepeer/issues/1966
- https://github.com/livepeer/go-livepeer/issues/2078
- https://github.com/livepeer/go-livepeer/issues/1757
- https://github.com/livepeer/go-livepeer/issues/2118
The question was raised on Discord to find out whether the autoAdjustPrice flag could help avoid the issue.
If I remember correctly, autoAdjustPrice allowed the node to adjust its PPP based on ETH price fluctuations, which could lead to job losses when the price changed significantly and the PPP exceeded the broadcaster’s set limit. However, this flag didn’t affect the face value of the ticket, which is what happens when gas prices rise. I don’t believe there’s currently a flag that can prevent this mechanism.
The question was raised on Discord to find out whether the autoAdjustPrice flag could help avoid the issue.
If I remember correctly, autoAdjustPrice allowed the node to adjust its PPP based on ETH price fluctuations, which could lead to job losses when the price changed significantly and the PPP exceeded the broadcaster’s set limit. However, this flag didn’t affect the face value of the ticket, which is what happens when gas prices rise. I don’t believe there’s currently a flag that can prevent this mechanism.
@FranckUltima thanks a lot for providing this extra information. You’re correct, these are two distinct issues that can lead to the same error. I haven’t had time for a deep dive yet, but it seems:
- One issue is related to
autoAdjustPrice, which triggers the error here when the ticket value exceeds the expected value. - The other occurs here when the face value is lower than the transaction cost.
Adjusted the priority level because this issue can be mitigated by gateways maintaining at least a 0.346 ETH reserve. Updated the documentation to reflect this recommendation (see PR #718. I however will try to find an owner to update the average gas estimation.
Cross posting this suggestion from @dob on Discord. I updated the issue to reflect this solution.
This comment may be out of date with the complexity of where all the sender reserve checks are handled in code, but I've long suggested it might be a good idea to add a startup flag on the O's that allow them to --ignoreSenderReserveRequirements and instead just check that their deposit is less than the ticket face value. Double spends don't seem to be a practical threat (at this stage anyway), and one of the main arguments for not increasing the # of O slots beyond 100 is that broadcaster reserves would have to increase.
If anyone cares about double spend prevention they can start their node without passing in that arg, but O's that are willing to take the risk can use that flag and then not worry about the sender reserves.