graph-node
graph-node copied to clipboard
Make `function not found` a deterministic failure
Do you want to request a feature or report a bug? Bug
What is the current behavior?
If a subgraph handler calls a function that doesn't exist, the subgraph fails with deterministic: false
, and the following error log:
failed to process trigger: block #13380099 (0x38d4…b3aa), transaction ca32b1581b837c6ca539cf6c2fd542e9bcacbdc345c6a424382b4f0e48b96153: function handleSwapped not found
What is the expected behavior? This handler will never run successfully, so this error should be deterministic
@otaviopace @leoyvens are there other errors known to be in the same category?
I've observed these myself:
(Same as OP) On block 13380098
(38d484acc0f3fe698ac05340877f5d8f766bb23f03883797a6e8452fb524b3aa
) subgraph QmWXF3jTLo6Wy9MxDZz6WKhET3GFcSFqn6vmrDDYPF7UHK
failed with
failed to process trigger: block #13380099 (0x38d4…b3aa), transaction ca32b1581b837c6ca539cf6c2fd542e9bcacbdc345c6a424382b4f0e48b96153: function handleSwapped not found
On block 13663246
(f5d44376510d09486cc9214cdf87be12948e6155ef76c8203fb963adde16c0f8
) subgraph QmY7xD8w9ekjrYLgSr7P1mNhi5dESzhnJmQjx3CssGZYVc
failed with
failed to process trigger: block #13663248 (0xf5d4…c0f8), transaction 7022d1ffc2b5bca80ef6c29115694f2c406b079d88c8957568e41d46c5fc6745: Entity Swap[0x7022d1ffc2b5bca80ef6c29115694f2c406b079d88c8957568e41d46c5fc6745-0]: missing value for non-nullable field `fromAmount` wasm backtrace: 0: 0x3172 - <unknown>!generated/schema/Swap#save 1: 0x360f - <unknown>!src/mappings/core/handleSwap
On block 13468096
(33939ae7b79fa2ec62390600d783cee53d344c3ceaed2a12c8237137248189c0
) subgraph QmZdsSbRwVD7VVVm5WGxZZC6HYvbjnFb4hcwvQ4fTs5bxA
failed with
failed to process trigger: block #13468099 (0x3393…89c0), transaction ee9aaf59819efc11375abe3c5ed0953597ef4daf805b4c0379bc81f1d232d155: Entity SavingsManager[0x9781c4e9b9cc6ac18405891df20ad3566fb6b301]: missing value for non-nullable field `address` wasm backtrace: 0: 0x18fe - <unknown>!generated/schema/SavingsManager#save 1: 0x2c38 - <unknown>!src/mappings/SavingsManager/handleStreamsFrozen
On block 4965314
(be4a6efc693c2e16b0df7ddeed5f057bd18e23544a3302b4efb7272aa3cbf066
) subgraph QmNrS2U5DHqn5DJiKKD1ZS4BpHeAGDsKjftHMwX8LqSZqv
failed with
failed to process trigger: block #4965316 (0xbe4a…f066), transaction ad66aefe198276fd6036ac7ae74d39cc75890f75a8d0456bb373fa92a38e8c73: Entity NFT[parcel-0xf87e31492faf9a91b02ee0deaad50d51d56d5d4d-14291859410679415465461733512134264881229]: missing value for non-nullable field `tokenId` wasm backtrace: 0: 0x183c - <unknown>!src/entities/schema/NFT#save 1: 0x1964 - <unknown>!src/handlers/parcel/handleUpdate
Two error variants observed:
-
function handleSwapped not found
-
missing value for non-nullable field
Semi related issue from my end on the fantom network.
Subgraph failed for non-deterministic error: failed to process trigger: block #20389388 (0x0000…d928), transaction c601d46c1b47732452b3554b4e12df4da307564d335feaff40b39d78372887e4: Decoding function outputs for the call Function { name: "rebase", inputs: [Param { name: "profit_", kind: Uint(256) }, Param { name: "epoch_", kind: Uint(256) }], outputs: [Param { name: "", kind: Uint(256) }], constant: false, state_mutability: NonPayable } failed, raw output: : Invalid name: please ensure the contract and method you're calling exist! failed to decode empty bytes. if you're using jsonrpc this is likely due to jsonrpc returning `0x` in case contract or method don't exist, retry_delay_s: 960, attempt: 3
It seems to be failing sporadically as not every failed transaction seems to trigger this error, but it is constantly failing on the same transactions.
Is there any way to bypass or work around this error/transaction?
@colonelssecretspices That would be a new issue affecting decoding call triggers, this issue is about handlers that do not exist in the mapping.
deployed with this https://github.com/impactMarket/subgraph/pull/15 and failed https://thegraph.com/hosted-service/subgraph/impactmarket/subgraph?version=pending also the "current version" is fully synced on celo mainnet with everything else but the cUSD contract. It's really weird (to me) that only fails with that contract.
Subgraph failed for non-deterministic error: failed to process trigger: block #2643840 (0x5f17…42cc), transaction 5f17d007958179510c0f73d411f8f3cbdcd2fa6201b554a9a76862e9cbf042cc: Mapping terminated before passing in trigger: send failed because receiver is gone, retry_delay_s: 1800```
Tried help on discord, for a few days, a couple of solutions, no solution yet.
Hi @obernardovieira and @colonelssecretspices , these seem to be different issues (call trigger decoding, and I'm not sure what the send failed because receiver is gone
error is @leoyvens?)
Hi @azf20 thanks for your reply. I'm available for anything I can be helpful with. The error happens when syncing the cUSD contract (both celo mainnet and celo alfajores). Although it only fails on the first empty block, until there, it actually registers a few entities. But If I remove the cUSD contract from the schema, then it doesn't fail. The "current version" is synced. And I tried with an empty handler for the cUSD contract, as mentioned above. So I think the error isn't on my end. Tried to get some help on discord but no success.
Adding to the list of fatalities that cannot be closed with the agent yet:
Opyn
On block 12,731,258
hash a1ab664e7213261d7f73bb9b44fca69dbe7d22ce9c600631bc3647a9d31faa00
id QmNRkaVUwUQAwPwWgdQHYvw53A5gh3CP3giWnWQZdA2BTE
failed with
Mapping aborted at generated/schema.ts, line 589, column 4, with message: Cannot save Vault entity without an ID wasm backtrace: 0: 0x17df - <unknown>!<wasm function 79> 1: 0x224b - <unknown>!<wasm function 114> in handler `handleShortOtokenMinted` at block #12731258 (a1ab664e7213261d7f73bb9b44fca69dbe7d22ce9c600631bc3647a9d31faa00)
@obernardovieira After #2797 the error message should become more specific. If you could reproduce again and send me the subgraph id on Discord I can try to look into the cause.
Another failure that should be deterministic:
On block 14043344
(5662c33b8e598339ced77eeb6bf4e58061528192bbaf907b309cc7fd6bfebee5
) subgraph QmRA6mjFXCmzYp5MwHzyHpUVehG6zNuL9xCw6oj4VD9HyZ
failed with
failed to process trigger: block #14043345 (0x5662…bee5), transaction 4499dc75facb4cd27894196aac00bcb0ffbe3d64de87105878451d5637662f90: Generating function inputs for the call Function { name: "atomicMatch_", inputs: [Param { name: "addrs", kind: FixedArray(Address, 14) }, Param { name: "uints", kind: FixedArray(Uint(256), 18) }, Param { name: "feeMethodsSidesKindsHowToCalls", kind: FixedArray(Uint(8), 8) }, Param { name: "calldataBuy", kind: Bytes }, Param { name: "calldataSell", kind: Bytes }, Param { name: "replacementPatternBuy", kind: Bytes }, Param { name: "replacementPatternSell", kind: Bytes }, Param { name: "staticExtradataBuy", kind: Bytes }, Param { name: "staticExtradataSell", kind: Bytes }, Param { name: "vs", kind: FixedArray(Uint(8), 2) }, Param { name: "rssMetadata", kind: FixedArray(FixedBytes(32), 5) }], outputs: [], constant: false, state_mutability: Payable } failed, raw input: ab834bab0000000000000000000000007be8076f4ea4a4ad08075c2508e481d6c946d12b000000000000000000000000f700b3d7b525f70092fc74128f8d578a683f25e90000000000000000000000008f0964189b28cd164ba1080a53962a26adafa51d0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000d4c3c23349348fa67f6d044f1357c54d8e6a661b000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007be8076f4ea4a4ad08075c2508e481d6c946d12b0000000000000000000000008f0964189b28cd164ba1080a53962a26adafa51d00000000000000000000000000000000000000000000000000000000000000000000000000000000000000005b3256965e7c3cf26e11fcaf296dfc8807c01073000000000000000000000000d4c3c23349348fa67f6d044f1357c54d8e6a661b0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003e8000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008e1bc9bf04000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000061e9844e0000000000000000000000000000000000000000000000000000000000000000dcae38d74061df89ea2ea74cb9dde60c668434e07bf7807470b44c2404c4324900000000000000000000000000000000000000000000000000000000000003e8000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008e1bc9bf04000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000061e639ad0000000000000000000000000000000000000000000000000000000061ea2e83bf8f6c924869e67ab3fa63401173f12706640cb32f9dde705fe82f9734f0e4fd0000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006a0000000000000000000000000000000000000000000000000000000000000074000000000000000000000000000000000000000000000000000000000000007e0000000000000000000000000000000000000000000000000000000000000088000000000000000000000000000000000000000000000000000000000000009200000000000000000000000000000000000000000000000000000000000000940000000000000000000000000000000000000000000000000000000000000001c000000000000000000000000000000000000000000000000000000000000001ce355fbc07931342c516655c51dd6d2632cc5ff92dd586bee0e9be17d030e260e7aa6e9f7837e6330ad097fcbd8b1dbc6fda1782dda2129ce6386e8bcaf921c96e355fbc07931342c516655c51dd6d2632cc5ff92dd586bee0e9be17d030e260e7aa6e9f7837e6330ad097fcbd8b1dbc6fda1782dda2129ce6386e8bcaf921c960000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006423b872dd0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f700b3d7b525f70092fc74128f8d578a683f25e9000000000000000000000000000000000000000000000000000000000000227e00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006423b872dd0000000000000000000000008f0964189b28cd164ba1080a53962a26adafa51d0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000227e00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006400000000ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000000000000000000000000000000000000000000000000000000000000000ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000: Invalid data
An instance of Generating function inputs for the call failed
.
Looks like the Generating function inputs for the call failed
variant has been fixed in the 0.25.1 release: https://github.com/graphprotocol/graph-node/issues/3194
@tilacog
Plan of attack:
- If a handler does not exist but the subgraph tries to call it -> make that a deterministic error.
- Add validation for the next subgraph API version that catches situations where handlers referenced in the manifest do not actually exist in the WASM mappings. If that's the case, reject the deployment or fail to start the subgraph.
Hello, here's an instance of mapping abort when saving an entity without ID
Block #13577690 (c4b62c0f23e1b41a32732f9fa91f2ed6d15d1bf6c7f0200ad7e6ab62ee681905)"
Subgraph QmViCANJ6HhQtFz6j3hmSLjVw5G5QqZrQQtNGB6pkugCyH
Mapping aborted at src/types/schema.ts, line 350, column 4, with message: Cannot save OfferParams entity without an ID\twasm backtrace:\t 0: 0x12ac - <unknown>!src/types/schema/OfferParams#save\t 1: 0x12ee - <unknown>!src/koans-offer/handleOfferDurationBlocksUpdated\t in handler `handleOfferDurationBlocksUpdated`
Another one, QmT2Y7SHXGRt7EKXzFPYMjnrgX64PQr86EjbsHoDLNzwLy
failed to process trigger: block #14528346 (0xc584…e112), transaction 21ecd1c9dd09371f03ef72eb28c3d524c956593d792107795d455af115c338e3: Entity Domain[0xb2aaf523bcf90507acc6937470253849686e2199ce45de9e04b3357e803accdd]: missing value for non-nullable field `owner` wasm backtrace: 0: 0x2125 - <unknown>!src/types/schema/Registration.load 1: 0x21a9 - <unknown>!src/ethRegistrar/handleNameRenewedByController
Another, QmZucDSFfuTNLDau2a3yFtt9wATXUKjLZ2ByX5fYE5Ps33
:
failed to process trigger: block #15840294 (0xb545…e2c3), transaction daee2857a95ec9096926a4fb5e3276aa2ffd4871b4a031cb22c00c4823a0e2ff: Entity Task[0xb6cb89194cfcb788c65763d9394b2f937046c7a13817db8754c72d9acb6e9190]: missing value for non-nullable field `version`\twasm backtrace:\t 0: 0x1ce2 - <unknown>!src/entities/schema/Task#save\t 1: 0x3726 - <unknown>!src/handlers/PokeMe/handleTimerSet\t