Sometimes operations are ignored and confirmation() fails with "confirmation polling timed out"

[852Kerfunkle]

Hello,

Basically, sometimes operations don't seem to be accepted by rpc nodes and they time out with "confirmation polling timed out" after some minutes.

Steps To Reproduce The easiest way to reproduce this seems to be to send an operation with insufficient gas (manually reduce the gas limit in temple to 20% of the estimate):

const batch_op = await some_wallet_operation_batch.send();
const op_res = await batch_op.confirmation(); // throws "confirmation polling timed out"
const op_status = await batch_op.status();
// check status etc

Expected behavior I'd expect to get some kind of error when the operation is rejected by the rpc node. (I think that's what's happening here - not sure).

Desktop (please complete the following information):

• OS: Linux
• Browser: Chromium
• Version: 12.1

[claudebarde]

Most of the time, when this error happens, it means that the node took longer than expected to send the confirmation. You can change the timeout value this way:

const Tezos = new TezosToolkit(RPC_URL);
Tezos.setProvider({
    config: { streamerPollingIntervalMilliseconds: VALUE_IN_MILLISECONDS }
});

in order to give more time to Taquito to wait for a response.

[852Kerfunkle]

@claudebarde I assume you meant confirmationPollingTimeoutSecond?

Thing is, the operation is never even added to the mempool (I think) - so I could set the timeout to hours and it would still happen.

If it seems convoluted to force a failure by setting the gas way too low, it's just a way I managed to to find to reproduce this behaviour. It does seem to happen in the wild - but it's difficult to reproduce. It's conceivable increasing the timeout might help there, I suppose.

[Innkst]

@852Kerfunkle do you still experience this issue or configuring timeout helped? Thanks

[852Kerfunkle]

@Innkst I just assumed the timeout would make it fail faster, but I feel that's just a workaround for the underlying issue here.

I didn't look at the details, but I would assume (maybe incorrectly) the node would give some kind of error when it plainly rejects an operation - it certainly does sometimes, just not in this case. I don't know if this is an RPC node issue or a taquito issue, but it sure is an issue that I occasionally experience under "normal" circumstances.

edit: or maybe a wallet issue? edit2: this happens with temple wallet.

[852Kerfunkle]

Oh, by the way - if my memory serves me right, it's also reproducible in a flextesa sandbox.

[OjusWiZard]

Hi everyone, I have been facing this error at every run in my code. It seems like I've found a way to recreate this issue. I've created a script here to reproduce this error, hoping that someone can help resolve this issue.

Edit:

• Versions:
  • ts-node v10.9.1
  • node v16.19.0
  • compiler v5.1.3
• OS: Ubuntu 22.04.2 LTS

[dvkam]

I am experiencing this odd behaviour too. I started the shifumi game from Marigold and when I try to submit a tx with Temple it always happens (I switched rpc too) , just like 852Kerfunkle said Thing is, the operation is never even added to the mempool (I think) - so I could set the timeout to hours and it would still happen. I experience the same. However when I switch to Kukai it works.

[hui-an-yang]

Hello @852Kerfunkle Thank you for bringing this up to us, we're looking to understand it better and hopefully resolve this issue. ~~I reproduced this bug by decreasing the gas_limit like you mentioned and it turned out that Kuaki is returning a insufficient_gas_for_manager error which is expected behaviour but temple do just hang there and can't resolve. Form the first look, it can be how temple is dealing with pre-apply operation error response didn't resolve it or reflect the error to user. (might need to look into their code to confirm You also mentioned that it can be reproduce in flextesa can you provide more context for us to investigate?~~

[852Kerfunkle]

Hello @hui-an-yang,

thank you for looking into this. I suppose it makes a lot of sense a mishadling of the pre-apply response in temple could be the root cause, I wasn't quite sure where to report this issue as I didn't have much time to investigate it myself.

Regarding flextesa, I only threw it in there as a pointer for debugging the issue, not that I thought the error is in the tezos node itself. I was using flextesa with Temple as well. Pretty much just used it as a local sandbox network and deployed some contracts to it.

Thanks!

[hui-an-yang]

Hello @852Kerfunkle, @dvkam, @OjusWiZard

I've done more investigation on this that I realized the temple Gas fee that let you adjust value of is actually fee not gasLimit. I tried even with octez-client when it comes to low fee operations they can get injected but was never confirmed. Here's the documentation of how without minimum fee the operation won't propagate into a block. If you configure your TezosToolkit with marigold rpc url and tried to reproduce this behaviour that you'll see the operations in the reject section of mempool_operation . We'll appreciate that you try this out and let us know if this finding will resolve the issue many people have :)

Side note that Kukai seem to enforce the fee that they calculated for users even that you tried to change in advanced mode it doesn't show in operations object at all.

Looking forward to hearing back from you

[852Kerfunkle]

Hey @hui-an-yang,

thanks a lot! Just a couple of question for my understanding:

The operation is rejected early (in this case, because of impossibly low fees - but I've seen this happen "in the wild" on "valid?" operations, just was tricky to reproduce) and the rejection does not propagate back to taquito, for some reason, and it waits on a confirmation until it times out. Correct?

Will this behaviour be fixed, that certain kinds of rejections will result in an error (one that doesn't involve waiting several minutes, I mean :) )?

[hui-an-yang]

Hi @852Kerfunkle

Thanks for replying, those are great question! Screenshot below is a list of reasons that an operation won't propagate into a block, I believe those you saw in the wild will belong to one of them. For the fact that the accessibility of a mempool is node dependent, it's a technical difficulty for taquito to returned the refused message and abort the search early for now. (don't think it's reasonable to only abort the search for those rpc node have mempool public access either) I have raised this behaviour to tezos core developer team to see if they're willing to improve on a protocol level. Will keep you updated when I hear back from them, hope this is helpful :)

[852Kerfunkle]

For the fact that the accessibility of a mempool is node dependent

I see. That does make it complicated.

Will keep you updated when I hear back from them, hope this is helpful :)

Very, thank you :)

Also, congratulations for getting to the bottom of this one!

edit: in the wild, I'm guessing it most likely was related to multiple manager operations unluckily landing on the same block, or the mempool being full.