User data export / (partial) import metaissue

[jaywink]

So to possibly get things moving, I'm creating here an issue for everything needed for user data export / (partial) import to happen. We've got good experience with splitting large items into independant tasks in issues like this - so hopefully this will help things forward and some day there will be a working export / (partial) import feature. I say partial because until the migration thingy is reality, it's not really feasible to import posts - but profile data should be no issue.

Note! Please contribute general discussion relating to this on Loomio here: https://www.loomio.org/d/w6nfQPFc/personal-data-import-export

Please feel free to work on single items and suggest smaller tasks in addition to these if necessary.

User data - export

• [x] Add user data export in JSON format (proposal, proposal) #5354
• [x] Add contacts, aspects and followed tags to data export #5354
• [x] Remove old XML export

User data - import

• [ ] Import for user data (profile, settings, etc, as exported)
• [ ] Import for contacts (automatic creation of aspects and adding contacts)

User content - export

• [x] Add posts and comments export to the user data export (pull #5499 )
• [x] Add/fix photos export tool (pull #5685)

[Faldrian]

It might be a different issue... but user migration from one pod to the other would still be a much needed feature. If I do this by exporting and importing a user, I have a a user account with contacts but none of my previous posts. Maybe it would be good to have some migration via federation, so the origin pod can issue a "user migrated to [email protected]" command and every pod with federated posts can rewrite that user information to match the new location. This way the former structure keeps intact. :) (This needs an existing target-user-account with some kind of auth-codes between the two pods involved in the migration... maybe inspired by internet domain transfer)

(I will remove my comment here if there is a place where it's better suited... I think there is, but I have no overview on where which discussion regarding this topic is happening.)

[jaywink]

@Faldrian you're thinking of #908 .

But slightly related - if account backup and restore ie migration was a real thing - do we really need an import for the JSON exports? The exports are needed as they are anyway, but importing them kinda becomes irrelevant imho.

[Faldrian]

@jaywink I have no clue what this was / is all about... it's a year since my comment above, I have forgotten everything and have no intention to re-read the topic... so you have a plan, do stuff, make us happy :)

[jaywink]

The latest automatic back up / restore spec proposal includes the import parts in this issue. Would be fancy to get more comments on it, split it into smaller chunks and dump a lot of bounty money on the issues - maybe get something to happen :)

Anyone wanting to give their insight, please check the proposal and comment in issue #908 . If no comments are received after some time I could just create a proposal to accept the proposal as a guideline for development.

[tclaus]

With #8274 this might be the missing "Seed wizard" ?

[dredmorbius]

Include canonical post URL in JSON data exports.

An issue I've repeatedly encountered with Diaspora* data export is that the provided JSON format does not provide either the short or the canonical URL for the post in question, nor so far as I can tell, any means for determining what that URL might be / have been.

Whilst the export does provide the content, what it does not provide is a means to supply a list of URLs to an archival service such that a given profile might be archived elsewhere online, the prime examples being at the Internet Archive Wayback Machine (IAWBM) via its "save URL now" URL (https://web.archive.org/save/<URL>).

The ability to do just that was critically useful during the shutdown of Google+, which otherwise offered a similar data export mechanism. It's proving a critical stumbling point with the recently announced planned shutdown of one of the oldest and largest Diaspora* instances, Joindiaspora.com, of which I've been a 10-year-plus active member.

REQUEST:: Add post URL to the JSON data export, preferably the long-form version of that URL.

E.g.,

• DO include the long form: https://joindiaspora.com/posts/b4fae69034500139f93f002590d8e506
• Do NOT include the short form: https://joindiaspora.com/posts/19830343

[Flaburgan]

Hello @dredmorbius, the JSON export contains the GUID of every element, if you want to rebuild the link, you just have to add https://joindiaspora.com/posts before the post GUID.

[dredmorbius]

Thanks, I'll look into that.

I need to wrestle with the JSON and jq parsing. (I've done that before, it's always a bit of an adventure.)

-- Edward Morbius @.***) Dr. Edward Morbius's Lair of the Id https://dredmorbius.reddit.com

Sent from ProtonMail mobile

-------- Original Message -------- On Jan 11, 2022, 02:07, Fla wrote:

Hello @.***(https://github.com/dredmorbius), the JSON export contains the GUID of every element, if you want to rebuild the link, you just have to add https://joindiaspora.com/posts before the post GUID.

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.Message ID: @.***>

[FiXato]

Thanks, I'll look into that. I need to wrestle with the JSON and jq parsing. (I've done that before, it's always a bit of an adventure.)

This one-liner should give you the combined post URLs for every one of your posts from your username_data.json archive: jq -r '.user .posts[] .entity_data | "https://REPLACE_WITH_YOUR_INSTANCE_DOMAIN/posts/\(.guid )"' username_data.json

If you want to include the created_at and text of each post: jq '.user .posts[] .entity_data | "https://REPLACE_WITH_YOUR_INSTANCE_DOMAIN/posts/\(.guid) \(.created_at) \(.text)"' username_data.json