twitter-to-sqlite
twitter-to-sqlite copied to clipboard
favorites --stop_after=N stops after min(N, 200)
For any number greater than 200, favorites --stop_after
stops after getting 200 tweets, e.g.
$ twitter-to-sqlite favorites tweets.db --stop_after=300
Importing favorites [####################################] 199
$
I don't think this is a limitation of the API (if you omit --stop_after
you get some very large number, possibly all of them), so I think this is a bug.
There's probably a nicer way of doing (hence this is a comment rather than a PR), but this appears to fix it:
--- a/twitter_to_sqlite/utils.py
+++ b/twitter_to_sqlite/utils.py
@@ -181,6 +181,7 @@ def fetch_timeline(
args["tweet_mode"] = "extended"
min_seen_id = None
num_rate_limit_errors = 0
+ seen_count = 0
while True:
if min_seen_id is not None:
args["max_id"] = min_seen_id - 1
@@ -208,6 +209,7 @@ def fetch_timeline(
yield tweet
min_seen_id = min(t["id"] for t in tweets)
max_seen_id = max(t["id"] for t in tweets)
+ seen_count += len(tweets)
if last_since_id is not None:
max_seen_id = max((last_since_id, max_seen_id))
last_since_id = max_seen_id
@@ -217,7 +219,9 @@ def fetch_timeline(
replace=True,
)
if stop_after is not None:
- break
+ if seen_count >= stop_after:
+ break
+ args["count"] = min(args["count"], stop_after - seen_count)
time.sleep(sleep)
This seems to be an issue even with larger values of --stop_after
:
$ twitter-to-sqlite favorites twitter.db --stop_after=2000
Importing favorites [####################################] 198
$