Tons of `Invalid Ticket senderNonce` for the past few months

[stronk-dev]

There seems to be tons of Invalid Ticket senderNonce coming in, especially in North America: [1] https://discord.com/channels/423160867534929930/932724294230900776/1024598556784463902 [2] https://discord.com/channels/423160867534929930/932724294230900776/1023253242697764924

It seems to be affecting all Orchestrators, did something change in the Broadcaster config? Maybe related to changes to use secondary broadcasters rather than the primary ones

[Franck-UltimaRatio]

I can confirm that, it greatly increases the error rate on tickets received, especially in North America.

[stronk-dev]

Have to add that I have also had very few winning tickets lately, even though there have been plenty of work/tickets coming in (averaging 0.0153 ETH daily in tickets in the last 7d, but only one winning ticket of 0.012 Eth). Could this be related?

[leszko]

@Franck-UltimaRatio @stronk-dev Can you confirm that is still an issue?

[Franck-UltimaRatio]

Yes it s still an issue, this is the number of error during the last hour on our NYC node (linux ubuntu)

[stronk-dev]

Seem to be happening less than before, but still some of them in US only

[leszko]

Is there any way you can check the Brodcaster's ETH addresses these request are coming from?

[Franck-UltimaRatio]

19049 segment_rpc.go:101] manifestID=cd0dc392-2dc1-44b9-809d-dd8311eb330b seqNo=1 orchSessionID=0486463c sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E clientIP=143.244.61.193 error processing payment: invalid ticket senderNonce sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E nonce=4 highest=6

[leszko]

I looked into the issue slightly deeper and here are some info/thoughts:

1. These streams come from the Livepeer Inc. Broadcasters
2. The issue is probably related to the conflicting rand numbers generated by the rand() function
3. One of the guesses is that it could happen when two streams starts during the same block and are transcoded by the same O.

Without any additional context, I think fixing it could take some significant effort (I'd estimate it to 2-10 days), so I leave it for now.

CC: @yondonfu @thomshutt

[yondonfu]

The issue is probably related to the conflicting rand numbers generated by the

The invalid ticket senderNonce error is triggered on O when it receives a ticket with a senderNonce value that is less than or equal to the highest senderNonce value tracked for a set of valid ticket params advertised by O. O should advertise a unique set of ticket params that is used for each session (as a result the same B should be using different sets of ticket params per session with an O). As a B sends tickets to O for a session using a set of ticket params, B uses a monotonically increasing senderNonce value with each ticket sent to O for that session.

I suspect this error is being triggered because for some reason the B is sending tickets to O with out-of-order senderNonce values - one possibility is that B retries an older segment/ticket after O has already received a newer segment/ticket. As already mentioned, this will require additional investigation to understand the observed behavior in order to implement a fix.

[yondonfu]

Solution is being tracked in https://github.com/livepeer/go-livepeer/issues/2661