Make the agent fully reloadable

[arichiardi]

Hello @mccraigmccraig, I stumbled upon this very solid addition to twitter-api and started using it on a proof of concept.

I noticed that after I call the first time cancel-twitter-stream, I start receiving:

18:47:36.948 [clojure-agent-send-off-pool-5] WARN  twitter-streaming-client.impl - ignoring body from incomplete or failed request
18:47:36.949 [clojure-agent-send-off-pool-5] WARN  twitter-streaming-client.impl - protocol failure. failure-count 3. backing off 80000ms until 2016-06-29T01:48:56.948Z
18:47:36.951 [clojure-agent-send-off-pool-5] WARN  twitter-streaming-client.impl - http.async.client/Response{id: req-id__51333, status: #promise[{:status :ready, :val {:code 420, :msg "Enhance Your Calm", :protocol "HTTP/1.1", :major 1, :minor 1}} 0x7faf11cc], headers: #promise[{:status :ready, :val {:connection "close", :content-length "36", :content-type "text/html", :date "Wed, 29 Jun 2016 01:47:36 GMT", :server "tsa", :x-connection-hash "e62c00e283ba896df01f3349258783ff"}} 0x4d4de8ce], done: #promise[{:status :ready, :val true} 0x7863f850], error: #promise[{:status :pending, :val nil} 0x7f45a7e1]}

It looks like some state keeps the connection alive.

I am willing to investigate on this, the code is very readable (thanks!).

[mccraigmccraig]

investigation welcome - i'm not using twitter at the moment, and this project has received no love from me recently - i'm very happy for it to get some from others though :)

[arichiardi]

This is kind of weird, I define my state with mount's defstate ^{:on-reload :noop} and on clojure.tools.../refresh I don't see any tracing of the :start method. However, I still see:

12:24:19.951 [clojure-agent-send-off-pool-6] INFO  twitter-streaming-client.impl - starting request: twitter.api.streaming$statuses_filter@266f6b with params: (:oauth-creds #twitter.oauth.OauthCredentials{ ...yada yada...})

Which means that the request is sent somehow on the wire.

[arichiardi]

Ok the above problem might be in mount.

So the problem with reload-ability is actually in the fact that when I call cancel-twitter-stream and then create-twitter-stream too fast I receive a 420. Twitter sees this as a problem apparently:

Clients which break a connection and then reconnect frequently (to change query parameters, for example) run the risk of being rate limited.

So is my best bet to wait for 1 minute as suggested?

Back off exponentially for HTTP 420 errors. Start with a 1 minute wait and double each attempt. Note that every HTTP 420 received increases the time you must wait until rate limiting will no longer will be in effect for your account.