twitter-to-sqlite icon indicating copy to clipboard operation
twitter-to-sqlite copied to clipboard

favorites --stop_after=N stops after min(N, 200)

Open mikepqr opened this issue 3 years ago • 2 comments

For any number greater than 200, favorites --stop_after stops after getting 200 tweets, e.g.

$ twitter-to-sqlite favorites tweets.db --stop_after=300
Importing favorites  [####################################]  199
$

I don't think this is a limitation of the API (if you omit --stop_after you get some very large number, possibly all of them), so I think this is a bug.

mikepqr avatar Sep 11 '20 03:09 mikepqr

There's probably a nicer way of doing (hence this is a comment rather than a PR), but this appears to fix it:

--- a/twitter_to_sqlite/utils.py
+++ b/twitter_to_sqlite/utils.py
@@ -181,6 +181,7 @@ def fetch_timeline(
     args["tweet_mode"] = "extended"
     min_seen_id = None
     num_rate_limit_errors = 0
+    seen_count = 0
     while True:
         if min_seen_id is not None:
             args["max_id"] = min_seen_id - 1
@@ -208,6 +209,7 @@ def fetch_timeline(
             yield tweet
         min_seen_id = min(t["id"] for t in tweets)
         max_seen_id = max(t["id"] for t in tweets)
+        seen_count += len(tweets)
         if last_since_id is not None:
             max_seen_id = max((last_since_id, max_seen_id))
             last_since_id = max_seen_id
@@ -217,7 +219,9 @@ def fetch_timeline(
                 replace=True,
             )
         if stop_after is not None:
-            break
+            if seen_count >= stop_after:
+                break
+            args["count"] = min(args["count"], stop_after - seen_count)
         time.sleep(sleep)

mikepqr avatar Sep 11 '20 04:09 mikepqr

This seems to be an issue even with larger values of --stop_after:

$ twitter-to-sqlite favorites twitter.db --stop_after=2000
Importing favorites  [####################################]  198
$

bcongdon avatar Sep 12 '20 14:09 bcongdon