Make `function not found` a deterministic failure

[azf20]

trafficstars

Do you want to request a feature or report a bug? Bug

What is the current behavior? If a subgraph handler calls a function that doesn't exist, the subgraph fails with deterministic: false, and the following error log:

failed to process trigger: block #13380099 (0x38d4…b3aa), transaction ca32b1581b837c6ca539cf6c2fd542e9bcacbdc345c6a424382b4f0e48b96153: function handleSwapped not found

Example subgraph

What is the expected behavior? This handler will never run successfully, so this error should be deterministic

[azf20]

@otaviopace @leoyvens are there other errors known to be in the same category?

[chriswessels]

I've observed these myself:

(Same as OP) On block 13380098 (38d484acc0f3fe698ac05340877f5d8f766bb23f03883797a6e8452fb524b3aa) subgraph QmWXF3jTLo6Wy9MxDZz6WKhET3GFcSFqn6vmrDDYPF7UHK failed with

failed to process trigger: block #13380099 (0x38d4…b3aa), transaction ca32b1581b837c6ca539cf6c2fd542e9bcacbdc345c6a424382b4f0e48b96153: function handleSwapped not found

On block 13663246 (f5d44376510d09486cc9214cdf87be12948e6155ef76c8203fb963adde16c0f8) subgraph QmY7xD8w9ekjrYLgSr7P1mNhi5dESzhnJmQjx3CssGZYVc failed with

failed to process trigger: block #13663248 (0xf5d4…c0f8), transaction 7022d1ffc2b5bca80ef6c29115694f2c406b079d88c8957568e41d46c5fc6745: Entity Swap[0x7022d1ffc2b5bca80ef6c29115694f2c406b079d88c8957568e41d46c5fc6745-0]: missing value for non-nullable field `fromAmount` wasm backtrace: 0: 0x3172 - <unknown>!generated/schema/Swap#save 1: 0x360f - <unknown>!src/mappings/core/handleSwap

On block 13468096 (33939ae7b79fa2ec62390600d783cee53d344c3ceaed2a12c8237137248189c0) subgraph QmZdsSbRwVD7VVVm5WGxZZC6HYvbjnFb4hcwvQ4fTs5bxA failed with

failed to process trigger: block #13468099 (0x3393…89c0), transaction ee9aaf59819efc11375abe3c5ed0953597ef4daf805b4c0379bc81f1d232d155: Entity SavingsManager[0x9781c4e9b9cc6ac18405891df20ad3566fb6b301]: missing value for non-nullable field `address` wasm backtrace: 0: 0x18fe - <unknown>!generated/schema/SavingsManager#save 1: 0x2c38 - <unknown>!src/mappings/SavingsManager/handleStreamsFrozen

On block 4965314 (be4a6efc693c2e16b0df7ddeed5f057bd18e23544a3302b4efb7272aa3cbf066) subgraph QmNrS2U5DHqn5DJiKKD1ZS4BpHeAGDsKjftHMwX8LqSZqv failed with

failed to process trigger: block #4965316 (0xbe4a…f066), transaction ad66aefe198276fd6036ac7ae74d39cc75890f75a8d0456bb373fa92a38e8c73: Entity NFT[parcel-0xf87e31492faf9a91b02ee0deaad50d51d56d5d4d-14291859410679415465461733512134264881229]: missing value for non-nullable field `tokenId` wasm backtrace: 0: 0x183c - <unknown>!src/entities/schema/NFT#save 1: 0x1964 - <unknown>!src/handlers/parcel/handleUpdate

Two error variants observed:

1. function handleSwapped not found
2. missing value for non-nullable field

[colonelssecretspices]

Semi related issue from my end on the fantom network.

Subgraph failed for non-deterministic error: failed to process trigger: block #20389388 (0x0000…d928), transaction c601d46c1b47732452b3554b4e12df4da307564d335feaff40b39d78372887e4: Decoding function outputs for the call Function { name: "rebase", inputs: [Param { name: "profit_", kind: Uint(256) }, Param { name: "epoch_", kind: Uint(256) }], outputs: [Param { name: "", kind: Uint(256) }], constant: false, state_mutability: NonPayable } failed, raw output: : Invalid name: please ensure the contract and method you're calling exist! failed to decode empty bytes. if you're using jsonrpc this is likely due to jsonrpc returning `0x` in case contract or method don't exist, retry_delay_s: 960, attempt: 3

It seems to be failing sporadically as not every failed transaction seems to trigger this error, but it is constantly failing on the same transactions.

Is there any way to bypass or work around this error/transaction?

[leoyvens]

@colonelssecretspices That would be a new issue affecting decoding call triggers, this issue is about handlers that do not exist in the mapping.

[vibern0]

deployed with this https://github.com/impactMarket/subgraph/pull/15 and failed https://thegraph.com/hosted-service/subgraph/impactmarket/subgraph?version=pending also the "current version" is fully synced on celo mainnet with everything else but the cUSD contract. It's really weird (to me) that only fails with that contract.

Subgraph failed for non-deterministic error: failed to process trigger: block #2643840 (0x5f17…42cc), transaction 5f17d007958179510c0f73d411f8f3cbdcd2fa6201b554a9a76862e9cbf042cc: Mapping terminated before passing in trigger: send failed because receiver is gone, retry_delay_s: 1800```

Tried help on discord, for a few days, a couple of solutions, no solution yet.

[azf20]

Hi @obernardovieira and @colonelssecretspices , these seem to be different issues (call trigger decoding, and I'm not sure what the send failed because receiver is gone error is @leoyvens?)

[vibern0]

Hi @azf20 thanks for your reply. I'm available for anything I can be helpful with. The error happens when syncing the cUSD contract (both celo mainnet and celo alfajores). Although it only fails on the first empty block, until there, it actually registers a few entities. But If I remove the cUSD contract from the schema, then it doesn't fail. The "current version" is synced. And I tried with an empty handler for the cUSD contract, as mentioned above. So I think the error isn't on my end. Tried to get some help on discord but no success.

[cryptovestor21]

Adding to the list of fatalities that cannot be closed with the agent yet:

Opyn On block 12,731,258 hash a1ab664e7213261d7f73bb9b44fca69dbe7d22ce9c600631bc3647a9d31faa00 id QmNRkaVUwUQAwPwWgdQHYvw53A5gh3CP3giWnWQZdA2BTE

failed with Mapping aborted at generated/schema.ts, line 589, column 4, with message: Cannot save Vault entity without an ID wasm backtrace: 0: 0x17df - <unknown>!<wasm function 79> 1: 0x224b - <unknown>!<wasm function 114> in handler `handleShortOtokenMinted` at block #12731258 (a1ab664e7213261d7f73bb9b44fca69dbe7d22ce9c600631bc3647a9d31faa00)

[leoyvens]

@obernardovieira After #2797 the error message should become more specific. If you could reproduce again and send me the subgraph id on Discord I can try to look into the cause.

[chriswessels]

Another failure that should be deterministic:

On block 14043344 (5662c33b8e598339ced77eeb6bf4e58061528192bbaf907b309cc7fd6bfebee5) subgraph QmRA6mjFXCmzYp5MwHzyHpUVehG6zNuL9xCw6oj4VD9HyZ failed with

failed to process trigger: block #14043345 (0x5662…bee5), transaction 4499dc75facb4cd27894196aac00bcb0ffbe3d64de87105878451d5637662f90: Generating function inputs for the call Function { name: "atomicMatch_", inputs: [Param { name: "addrs", kind: FixedArray(Address, 14) }, Param { name: "uints", kind: FixedArray(Uint(256), 18) }, Param { name: "feeMethodsSidesKindsHowToCalls", kind: FixedArray(Uint(8), 8) }, Param { name: "calldataBuy", kind: Bytes }, Param { name: "calldataSell", kind: Bytes }, Param { name: "replacementPatternBuy", kind: Bytes }, Param { name: "replacementPatternSell", kind: Bytes }, Param { name: "staticExtradataBuy", kind: Bytes }, Param { name: "staticExtradataSell", kind: Bytes }, Param { name: "vs", kind: FixedArray(Uint(8), 2) }, Param { name: "rssMetadata", kind: FixedArray(FixedBytes(32), 5) }], outputs: [], constant: false, state_mutability: Payable } failed, raw input: ab834bab0000000000000000000000007be8076f4ea4a4ad08075c2508e481d6c946d12b000000000000000000000000f700b3d7b525f70092fc74128f8d578a683f25e90000000000000000000000008f0964189b28cd164ba1080a53962a26adafa51d0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000d4c3c23349348fa67f6d044f1357c54d8e6a661b000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007be8076f4ea4a4ad08075c2508e481d6c946d12b0000000000000000000000008f0964189b28cd164ba1080a53962a26adafa51d00000000000000000000000000000000000000000000000000000000000000000000000000000000000000005b3256965e7c3cf26e11fcaf296dfc8807c01073000000000000000000000000d4c3c23349348fa67f6d044f1357c54d8e6a661b0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003e8000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008e1bc9bf04000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000061e9844e0000000000000000000000000000000000000000000000000000000000000000dcae38d74061df89ea2ea74cb9dde60c668434e07bf7807470b44c2404c4324900000000000000000000000000000000000000000000000000000000000003e8000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008e1bc9bf04000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000061e639ad0000000000000000000000000000000000000000000000000000000061ea2e83bf8f6c924869e67ab3fa63401173f12706640cb32f9dde705fe82f9734f0e4fd0000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006a0000000000000000000000000000000000000000000000000000000000000074000000000000000000000000000000000000000000000000000000000000007e0000000000000000000000000000000000000000000000000000000000000088000000000000000000000000000000000000000000000000000000000000009200000000000000000000000000000000000000000000000000000000000000940000000000000000000000000000000000000000000000000000000000000001c000000000000000000000000000000000000000000000000000000000000001ce355fbc07931342c516655c51dd6d2632cc5ff92dd586bee0e9be17d030e260e7aa6e9f7837e6330ad097fcbd8b1dbc6fda1782dda2129ce6386e8bcaf921c96e355fbc07931342c516655c51dd6d2632cc5ff92dd586bee0e9be17d030e260e7aa6e9f7837e6330ad097fcbd8b1dbc6fda1782dda2129ce6386e8bcaf921c960000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006423b872dd0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f700b3d7b525f70092fc74128f8d578a683f25e9000000000000000000000000000000000000000000000000000000000000227e00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006423b872dd0000000000000000000000008f0964189b28cd164ba1080a53962a26adafa51d0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000227e00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006400000000ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000000000000000000000000000000000000000000000000000000000000000ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000: Invalid data

An instance of Generating function inputs for the call failed.

[chriswessels]

Looks like the Generating function inputs for the call failed variant has been fixed in the 0.25.1 release: https://github.com/graphprotocol/graph-node/issues/3194

[Jannis]

@tilacog

Plan of attack:

1. If a handler does not exist but the subgraph tries to call it -> make that a deterministic error.
2. Add validation for the next subgraph API version that catches situations where handlers referenced in the manifest do not actually exist in the WASM mappings. If that's the case, reject the deployment or fail to start the subgraph.

[hopeyen]

Hello, here's an instance of mapping abort when saving an entity without ID Block #13577690 (c4b62c0f23e1b41a32732f9fa91f2ed6d15d1bf6c7f0200ad7e6ab62ee681905)" Subgraph QmViCANJ6HhQtFz6j3hmSLjVw5G5QqZrQQtNGB6pkugCyH

Mapping aborted at src/types/schema.ts, line 350, column 4, with message: Cannot save OfferParams entity without an ID\twasm backtrace:\t    0: 0x12ac - <unknown>!src/types/schema/OfferParams#save\t    1: 0x12ee - <unknown>!src/koans-offer/handleOfferDurationBlocksUpdated\t in handler `handleOfferDurationBlocksUpdated` 

[chriswessels]

Another one, QmT2Y7SHXGRt7EKXzFPYMjnrgX64PQr86EjbsHoDLNzwLy

failed to process trigger: block #14528346 (0xc584…e112), transaction 21ecd1c9dd09371f03ef72eb28c3d524c956593d792107795d455af115c338e3: Entity Domain[0xb2aaf523bcf90507acc6937470253849686e2199ce45de9e04b3357e803accdd]: missing value for non-nullable field `owner` wasm backtrace: 0: 0x2125 - <unknown>!src/types/schema/Registration.load 1: 0x21a9 - <unknown>!src/ethRegistrar/handleNameRenewedByController

[chriswessels]

Another, QmZucDSFfuTNLDau2a3yFtt9wATXUKjLZ2ByX5fYE5Ps33:

failed to process trigger: block #15840294 (0xb545…e2c3), transaction daee2857a95ec9096926a4fb5e3276aa2ffd4871b4a031cb22c00c4823a0e2ff: Entity Task[0xb6cb89194cfcb788c65763d9394b2f937046c7a13817db8754c72d9acb6e9190]: missing value for non-nullable field `version`\twasm backtrace:\t    0: 0x1ce2 - <unknown>!src/entities/schema/Task#save\t    1: 0x3726 - <unknown>!src/handlers/PokeMe/handleTimerSet\t