Reconncting stream after 30 secs results in "420 Enhance your calm" error

[karthik947]

Reconncting stream after 30 secs results in "420 Enhance your calm" error.

My requirements is to update the stream filter, which requires me to disconnect and reconnect. Is there a way to update statuses/filter without disconnects ? If that is not possible, whats the recommended wait duration b/w reconnects.

Thanks for your input.

[karthik947]

After testing again, looks like stream.destroy() does not end the connection, which is why when I try to reconnect after 30 secs, I get the "Maximum conenctions exceeded" error.

[ImTheDeveloper]

Agree I'm also seeing this issue even though I'm definitely doing a stream destroy correctly.

I've found I have to restart my whole project container to clear the issue as no amount of back off/waiting after calling the destroy function works.

[karthik947]

Yeah, restarting the app works fine as opposed to destroy and re-establish, but restarting is not always possible and only works as a work around. I hope someone from the community responds with a permanent solution.

[dandv]

I suspect this might be a problem with Twitter. Have you checked if users of libraries for other languages have run into this?

[ImTheDeveloper]

Worked through this most of today and can not fathom why destroying the stream doesn't actually seem to fix the problem.

I can destroy and create immediate = enhance calm I can destroy, wait for 2 minutes and create = enhance calm I've extended this right up to 30 minutes and still persists.

With a process.exit and restart I can bypass the enhance calm right away. Very odd.

[jxlarrea]

I've been giving twitter-lite a try since TweetInvi, the library I've been using for a few years now, seems to be abandoned. After migrating my .net project to node, I also encountered the issue a lot of folks are having right now - streams randomly dying. I can confirm that this is not caused by twitter-lite or Node, since the same exact behavior is seen with TweetInvi in .NET.

Anyway, since I also have to deal with the whole stream dying / reconnecting thing, what I'm doing is within the stream.on("end") event I setup a 30 second setTimeout with process.exit(1). This strategy is working great - I haven't seen a single enhance your calm throttle so far.

  stream.on("end", response => {            
            console.error("End", "Stream Ended. Exiting in 30 seconds...");            
            setTimeout(function () {
                console.error("End", "Exiting now.");
                process.exit(1); 
            }, 30000);
            process.nextTick(() => stream.destroy());           
        });

[karthik947]

Chris was referring to the same. Using process.exit() terminates the application, which is not a graceful way to disconnect.

On Mon, Jul 6, 2020 at 8:53 PM Xavier Larrea [email protected] wrote:

I've been giving twitter-lite a try since TweetInvi, the library I've been using for a few years now, seems to be abandoned. After migrating my .net project to node, I also encountered the issue a lot of folks are having right now - streams randomly dying. I can confirm that this is not caused by twitter-lite or Node, since the same exact behavior is seen with TweetInvi in .NET.

Anyway, since I also have to deal with the whole stream dying / reconnecting thing, what I'm doing is within the stream.on("end") event I setup a 30 second setTimeout with process.exit(1). This strategy is working great - I haven't seen a single enhance your calm throttle so far.

stream.on("end", response => { console.error("End", "Stream Ended. Exiting in 30 seconds..."); setTimeout(function () { console.error("End", "Exiting now."); process.exit(1); }, 30000); process.nextTick(() => stream.destroy()); });

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/draftbit/twitter-lite/issues/118#issuecomment-654304347, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHFKOQ7FUNWSJCAFAUXKQULR2HUAFANCNFSM4OPA7IKA .

-- Best Regards, Karthik.

[jxlarrea]

Chris was referring to the same. Using process.exit() terminates the application, which is not a graceful way to disconnect.

It works for me, as my process runs in kubernetes which automatically recreates the pod after it dies when using process.exit(). I do agree however that this shouldn't be necessary.

[ImTheDeveloper]

I actually still get these sometimes even after all the fixes 😂 how is it going for everyone else?

I spent hours today looking across all libraries in other languages to get the general feel for how this is being handled across the board.

Outcome? Everyone seems to be losing their hair.

[tejazz]

Is there any update on the issue? I have been trying to resolve the same problem for a simple streaming application. Every time I try to stop a stream using stream.destroy(), it does not seem to end the connection and restarting the connection throws a 420 error.

[tejazz]

@dandv Any updates on this issue?

[rlueder]

Perhaps this section of the v1 API documentation can be helpful? https://developer.twitter.com/en/docs/twitter-api/v1/tweets/filter-realtime/guides/connecting#disconnections

Looking at how twitter-lite implements streams none of the stall or disconnect strategies are implemented and it's up to each application to implement their own.

Specifically to @karthik947 original post 30s is below what Twitter says it's the minimum time to wait before attempting to reconnect (3 cycles, i.e. 90s):

Set a timer, either a 90 second TCP level socket timeout, or a 90 second application level timer on the receipt of new data. If 90 seconds pass with no data received, including newlines, disconnect and reconnect immediately according to the backoff strategies in the next section.

Reading a bit more the documentation, there seems to be a "catch" specific to 420 errors:

Back off exponentially for HTTP 420 errors. Start with a 1 minute wait and double each attempt. Note that every HTTP 420 received increases the time you must wait until rate limiting will no longer will be in effect for your account.

From that sentence it seems that every 420 returned adds a full minute to the time needed to wait before the next attempt is made.

[tejazz]

Incremental waits is acceptable. Although makes it harder to provide the functionality to constantly change the track for status/filter. 🤦🏻‍♂️