forget icon indicating copy to clipboard operation
forget copied to clipboard

New component for uploading archives

Open codl opened this issue 3 years ago • 2 comments

Part of #64 Blocked by #458

  • Present itself as a large "Import archive" button, which opens a file picker and accepts either a tweet.js file or a zip file
  • Also a drop zone
  • If given a zip file, looks for a tweet.js file inside it. zip.js seems deece.
    • might be good to load zip.js on demand when a zip file is given
  • Be able to parse a tweet.js, extract the useful metadata for each tweet, and upload it in batches to the endpoint in #458
  • Some live progress reporting would be good

Also needs some copy (not inside the component) explaining what the archive is and does, and that the zip file won't be uploaded but processed locally. Some of the old copy can probably be reused.

I will not be making a no-JS plain-html alternative, because I absolutely do not want people to accidentally upload their whole archive to me. I don't want your DMs.

codl avatar May 14 '21 13:05 codl

Of note is that the same component could potentially support ActivityPub archives in the future. Though AP archives aren't quite as straightforward to work with.

codl avatar May 14 '21 14:05 codl

observation from the archive i ordered yesterday and got today:

I haven't seen an archive with more than one tweet.js file but there also exists a manifest.js file that seems to imply that more could exist, and that it would catalog them:

window.__THAR_CONFIG = {
  "dataTypes" : {
    "tweet" : {
      "mediaDirectory" : "data/tweet_media",
      "files" : [ {
        "fileName" : "data/tweet.js",
        "globalName" : "YTD.tweet.part0",
        "count" : "495"
      } ]
    }
  }
}

(truncated, there is much more in manifest.js)

e: i should also include the structure of tweet.js:

window.YTD.tweet.part0 = [ {
  "tweet" : {
    "retweeted" : false,
    "source" : "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>",
    "entities" : {
      "hashtags" : [ ],
      "symbols" : [ ],
      "user_mentions" : [ ],
      "urls" : [ ]
    },
    "display_text_range" : [ "0", "20" ],
    "favorite_count" : "0",
    "id_str" : "1364581575388708865",
    "truncated" : false,
    "retweet_count" : "0",
    "id" : "1364581575388708865",
    "created_at" : "Wed Feb 24 14:22:54 +0000 2021",
    "favorited" : false,
    "full_text" : "ah, my notifications",
    "lang" : "en"
  }
}, ... ]

far as i can tell, the contents of window.YTD.tweet.part0[x].tweet are identical to what the API would return. It may be better to send the raw tweet object to the server and let the existing code there deal with it.. but that would also mean sending a lot of extraneous data that the server doesn't need and would potentially make the import process bandwidth-bound

codl avatar May 15 '21 22:05 codl