twitter-archive-parser icon indicating copy to clipboard operation
twitter-archive-parser copied to clipboard

Extract collecting userids and do handle lookup in bulk

Open flauschzelle opened this issue 2 years ago • 1 comments

Implementation of this TODO for the other affected types of data (followings, followers, DMs).

The script now only asks 'Download user data from Twitter? [y/n]' once instead of three (or would have been four after merging the group DMs parsing, if that were done like the others before) times.

Here is an example snippet of the output from this version:

Parsing ./data/account.js...
Parsing ./data/tweets.js...
Wrote 27264 tweets to *.md and *.html, with images and video embedded from ./media
Parsing ./data/following.js...
found 704 user IDs in followings.
Parsing ./data/follower.js...
found 528 user IDs in followers.
Parsing ./data/direct-messages.js...
found 87 user IDs in direct messages.

488 users are unknown.
Download user data from Twitter (approx 1,024KB)? [y/n]y

Parsing ./data/following.js...
Wrote 704 accounts to ./following.txt
Parsing ./data/follower.js...
Wrote 528 accounts to ./followers.txt
Parsing ./data/direct-messages.js...

...

Wrote 87 direct message conversations (3860 total messages) to 89 markdown files

flauschzelle avatar Nov 25 '22 11:11 flauschzelle

If #118 is merged before this, I will adapt this to better integrate the changes from #118.

(And the reverse applies if this is merged before #118 - then I'll adapt #118 to make better use of this feature there.)

flauschzelle avatar Nov 25 '22 14:11 flauschzelle

The only issue with this is that someone with 40K followers might well want to retrieve the handles for their DMs but not sit there and download all the followers.

But let's treat that as a later bug to be solved. For example we might have a threshold - if there are more than 2K handles to download, then ask separately about DMs.

timhutton avatar Nov 26 '22 02:11 timhutton

The only issue with this is that someone with 40K followers might well want to retrieve the handles for their DMs but not sit there and download all the followers.

But let's treat that as a later bug to be solved. For example we might have a threshold - if there are more than 2K handles to download, then ask separately about DMs.

Or rather ask separately about followers, if that tends to be the thing that might take longer and would not be of use to some people?

flauschzelle avatar Nov 26 '22 12:11 flauschzelle