twitter-archive-parser
twitter-archive-parser copied to clipboard
timhutton
Feature request: Export unshortened URLs to CSV (e.g. for archiving them in the Internet Archive)
Michele Weigle has a nice thread on archiving t.co links into the Internet Archive running an Internet Archive service that archives URLs in a Google Sheets spreadsheet: https://twitter.com/weiglemc/status/1593698822257102851
Her script prepares the list of t.co URLs using awk: https://gist.github.com/weiglemc/312a11356420b3bc4c8e196e8454002a
The idea from that script might be a thing you want to include in your Python script.
Related: https://github.com/timhutton/twitter-archive-parser/pull/42 (via https://github.com/timhutton/twitter-archive-parser/issues/38#issuecomment-1312829030)
@jpluimers We do this already, in parse_tweets(). #42 is then taking the next step, which is to make remote calls to t.co (wp.me, etc.) to retrieve the expanded URLs directly for the ones we couldn't find in the archive.
What I mean is, we already get the expanded versions from the JSON. We don't push the mappings to the internet archive, though I can see that that is a useful service to humanity and something we could support.
What I mean is, we already get the expanded versions from the JSON. We don't push the mappings to the internet archive, though I can see that that is a useful service to humanity and something we could support.
That would be cool. What's the best way to rephrase this issue to reflect that intent better?
What I mean is, we already get the expanded versions from the JSON. We don't push the mappings to the internet archive, though I can see that that is a useful service to humanity and something we could support.
Yeah if there could be some sort of "export to Google Sheet" option to make the step outlined in the 3rd tweet of the thread linked in the OP easier, I would find that useful: https://twitter.com/weiglemc/status/1593698828171067393
