Create a data backup process

[aaronshaf]

I'd like to wipe out my application settings and reinstall... but I'd like to preserve my Dat archives. Is this possible?

[pfrazee]

We don't have an automated process yet. We should create one.

Let's give this a shot.

1. Find the the path to user data. It is indicated in your settings page. (For mac it's /Users/paulfrazee/Library/Application Support/Beaker Browser).
2. Move the entire user data directory to somewhere safe. Perhaps ~/beaker.backup
3. Start beaker. This will create a fresh and empty user data dir for you.
4. Close beaker.
5. Copy the Profiles file from your backup into your new empty user data dir, overwriting the new Profiles file.
6. Copy the Dat archive folders that you want to restore. You can find them in your backup, under Dat/Archives/Meta/. They are addressed in a funny way for performance -- first 2 chars of the pubkey make a folder, then the remaining chars make a subfolder.
7. Start beaker. You should now have all the dats you wanted to save.

Good luck. We'll get a tool for this soon, I hope.

[astra137]

I do not understand how Dat deals with dat ownership, but as best as I can tell using Beaker Browser, Dat Desktop, and dat CLI, you cannot make edits to a dat from anywhere except the initial dat folder. I tried simply moving the secret key out of Archives/Meta into ~/.dat/secret_keys, but it seems dat's know if they are the origin or a clone.

If it's possible, it would be cool to be able to export a site created in Beaker for other dat clients so that editing was still possible. I'll keep reading up on Dat in the meantime. Thank you for the walkthrough, @pfrazee!

[pfrazee]

@maccelerated yeah we need to make this easier. The reason we haven't yet is that it's possible to fatally corrupt your Dats right now if you write from multiple processes. Not good. I'm exploring ways we can fix that in the long term, but for now just be careful using this process.

[astra137]

Thank you for the warning.

[aaronshaf]

Is this issue solved with the shortly coming multi-author dat stuff? :) maf's stuff, etc

[pfrazee]

Multi-author (aka Multiwriter) will solve this problem. Rather than copying private keys between your devices, you'll add each device as an owner over the dats. That way corruptions cant occur and you have total control with each device.

[DougAnderson444]

@pfrazee I'm wondering what the latest multiwriter status is? I saw Jim Pick's Shopping List with the @mafintosh HyperDB mutli-writer updates ...just wondering how mature this all is now? I'd definitely want to be able to update Dat from anywhere also, not just the origin folder / machine. Thanks!

[pfrazee]

@DougAnderson444 That initial tech demo has been iterated a lot since then. There's going to be a string of tech updates over the next 6 months, and I expect multiwriter will land at the end of summer.

EDIT: this ended up being 100% wrong. We had to drop the multiwriter priority to focus on better scaling properties, so multiwriter is still in the future. On the plus side, Hyperdrive scales incredibly well now.

[DougAnderson444]

it's possible to fatally corrupt your Dats right now if you write from multiple processes.

Yikes. Until the updates are done, does that mean we should avoid using the process cited here for moving dats

(original) dat keys export - copy the secret key printed out.
(duplicate) dat keys import - this will prompt you for the secret key, paste it in here.

...or simply proceed with caution, as in, don't try to write from two places at the same time.

[pfrazee]

@DougAnderson444 It's a tricky situation. You need to avoid writing from 2 places at once, and if you restore from backups you risk overwriting history if your data isn't 100% up-to-date. One of our bigger issues.

[alexindigo]

Multi-author (aka Multiwriter) will solve this problem. Rather than copying private keys between your devices, you'll add each device as an owner over the dats

@pfrazee How about use case when your device has died, but you still posses private/public keys and want to restore your ownership. Would it be possible with multiwriter? Or what would be better approach? Thank you.

[pfrazee]

@alexindigo It would make a lot of sense to appoint backup nodes (home devices, cloud services, etc) which you can restore from. They would ensure you refetch all of the data before you write again, avoiding any potential for corruption.

[alexindigo]

@pfrazee I understand there are different avenues to solve certain problems, but in the same time there are use cases, where backup (data restore) outside of dat realm isn't available. For the example sake, imagine journalist with a burner phone somewhere in a remote country, their phone got repossessed and they did memorized their passphrase. And since their dat data available among their peers, it could be easily refetched to the new device, all they need is just to get back control over it. Just a hypothetical scenario :)

[pmario]

And since their dat data available among their peers, it could be easily refetched to the new device, all they need is just to get back control over it. Just a hypothetical scenario :)

If you can read the whole data, imo it's OK. You can clone the existing dat archive and you get a new access key. ... You just can't use the old dat address.

If you did a backup of the private key you can even restore access. So it's only 1 additional file, that you need to save to a safe place.

[pfrazee]

@alexindigo I don't think what I described runs counter to that. So long as you can fetch the latest state of the dat and restore the private key on your device, you're in business!

[alexindigo]

@pfrazee I see. Thanks a lot for the explanation. I got confused by initial comments, it looked like multi-writer is the only way to achieve "backup" functionality. Thank you.

[kriskeillor]

So I've copied Profiles into a backup folder, from both my new & old computers. I intend to replace the new with the old, but I have been using Beaker on my new browser, so I'm backing it up.

I'm also copying all of Archives/Meta right now. So once that's finished (windows just bumped the estimate up to 23 hours...), if I copy the old Profiles over to the new computer's Beaker Browser/ root folder, I'll have edit control? And the couple of dats I seeded on my new computer won't be overwritten by copying Meta's contents, so I'll still have those? They don't have any more peers except me. :(

Update: It totally worked! I again have edit control of my dats.

However, I lost the two dats I had that no-one else had. Is there a way to restore them from my backup? There's 10 folders in the Meta backup, but all of their names are already taken by my restored personal dats. I can't decypher this two-letter naming scheme. But it seems like the name was overwritten.

[pfrazee]

@krisradio We don't have an official process because currently the protocol doesn't support writes from 2 different devices. There's a chance you can corrupt the data if that happens, so just be sure not to run Beaker from the previous device.

We're looking at how to improve the protocol so that multiple devices won't corrupt the data.

[kriskeillor]

Previous device has been reformatted as a backup drive, so there's no chance of that.

I understand multiwriter support is still in progress, I just wanted to share my process for anyone else... I accept no liability!

[rhythnic]

For reference, here's a makefile for manually backing-up and restoring Beaker dats in Linux (I'm using Ubuntu), according to the process suggested above. I'm only backing up the Profiles file and the Dat/Archives/Meta dir, but I think that's enough. The backup file is compressed and encrypted. You'll be prompted for a password when you run the beaker_backup command.

Makefile

beaker_backup:
	mkdir -p /tmp/Beaker\ Browser/Dat/Archives
	cp ~/.config/Beaker\ Browser/Profiles /tmp/Beaker\ Browser/Profiles
	cp -r ~/.config/Beaker\ Browser/Dat/Archives/Meta /tmp/Beaker\ Browser/Dat/Archives/
	tar -cJC /tmp/ Beaker\ Browser | gpg -c -o ~/beaker-backup.tar.xz.gpg
beaker_backup_clean:
	rm -rf /tmp/Beaker\ Browser
beaker_restore:
	# first open then close beaker
	rm -rf /tmp/Beaker\ Browser
	gpg -d ~/beaker-backup.tar.xz.gpg | tar -xJC /tmp
	cp /tmp/Beaker\ Browser/Profiles ~/.config/Beaker\ Browser/Profiles
	cp -r /tmp/Beaker\ Browser/Dat/Archives/Meta/* ~/.config/Beaker\ Browser/Dat/Archives/Meta/
beaker_restore_clean:
	rm ~/beaker-backup.tar.xz.gpg
	rm -rf /tmp/Beaker\ Browser

Disclaimer

If you lose your password you'll lose your data. I'm not responsible for any data loss. :)

Backup Process

# cd to the directory with the makefile
make beaker_backup
make beaker_backup_clean

Backup the ~/beaker-backup.tar.xz.gpg file (USB, Google Drive, etc).

Restore Process

Make sure you've opened and closed Beaker to ensure the config folder exists. Download beaker-backup.tar.xz.gpg to your home directory (~/).

# cd to the directory with the makefile
make beaker_restore
make beaker_restore_clean

.tar.xz.gpg

The extension lets you know how the file is constructed. First the archive is created (.tar), then it's compressed (.xz), then it's encrypted (.gpg)

[johanbove]

The process of restoring with only the profiles sqlite database no longer seems to work in Beaker beta.7. When I start the app, it still starts with the "Create a profile" intermediate wizard. @pfrazee Is there a way to start Beaker making it think the profile was already set? I saw in the sqlite dbs some entries, but didn't manage to set the right flags I think.

Update: discovered where I can "trick" the setup-flow to think my profile was already set: https://github.com/beakerbrowser/beaker/blob/cbc7cbbc07d9634330d24b97a3875139f47f333e/app/bg/ui/setup-flow.js#L40