Unclear how to update a dat 'from scratch' / dat keys import doesn't notice a dat

[tmcw]

I am reporting:

• [ ] a bug or unexpected behavior
• [x] general feedback
• [ ] feature request
• [ ] security issue

Bug Report

Please give us details about your installation to assist you. Run dat -v to see the version of Dat you are using.

• Operating system: macOS High Sierra
• Node Version: 8.9
• Dat Version: 13.10.0

Expected behavior

dat keys import ... should import keys that allow you to write to a dat archive that you've cloned with dat clone

Actual behavior

I clone my dat like so:

dat clone dat://cff13e4cc797f46d453d4b2d7472dd481d7fd2f0a1c6e4eab425327f48c396fd _site

And then cd into the directory ./_site, and then try to import the key:

dat keys import af32293caa7....

And I get an error message:

  dat Dat DEBUG mode engaged, enabling quiet mode +0ms
  dat dat 13.10.0 +24ms
  dat node v8.9.4 +0ms
  dat dat keys +0ms
Sorry, could not find a dat in this directory.

But there is, quite certainly, a dat present:

_site〉ls .dat
content.bitfield    content.signatures  metadata.bitfield   metadata.key        metadata.signatures
content.key         content.tree        metadata.data       metadata.latest     metadata.tree

---

More broadly, I'm still struggling to understand how to solve the 'republishing' problem I noted in my blog post about dat. Basically:

• I have a website, and publish it from the _sites directory
• My computer is eaten by a bear
• How do I update my website

It seems like, given the dat://... link and the keys, I should be able to update it, and my guess is that I'd:

• use dat clone dat://... _sites
• run jekyll build to update the files in _sites
• use dat keys import ... to allow me to write to that
• use dat sync to publish updates

However, this has issues:

1. dat keys import doesn't seem to import my keys
2. What if there aren't any seeders / peers of my site, so I can't run dat clone because there's nobody to clone from?

I realize that I'm thinking about dat only occasionally, so maybe I'm looking at this from entirely the wrong angle - is there a different way to solve this problem that I'm not seeing?

[joehand]

Thanks for the awesome issue! Can reproduce the behavior. Apologies for having this command not be documented and thus impossible to use 😕 .

To import your key run dat keys import without any argument:

cd _sites
dat keys import

Dat will prompt for your key, paste your exported key into the terminal. I hope that solves it! Thanks for your patience.

Should look like:

Please keep this issue open so I can fix passing the key as an argument.

[tmcw]

Any idea about the other issue? Do dat URLs 'die' if they go to 0 seeders, or is there any way to re-share a dat given a URL, directory of data, and a key?

[pfrazee]

@tmcw Correct. If you lose your original but have the keys in cold storage, and wanted to restore the dat, you'd need a seeder to get the data from (and that seeder would need to have the latest data).

[rhodey]

I'm trying to use dat for backups and like @joehand I am also confused by the recommended pattern here. What's the difference between the key you import using $ dat keys import and the keys sitting in your ~/.dat/secret_keys directory?

I created an archive on HostA, then cloned it on HostB, then deleted it on HostA, then cloned it on HostA from HostB, and now the archive is not writable from HostA even though I never touched my ~/.dat/secret_keys directory.

[joehand]

Hi @rhodey, the key that sits in ~/.dat/secret_keys is what is printed when you run dat keys import. So importing/exporting manages keys in that folder.

To be writable you also need a metadata.ogd (original dat) in the .dat folder. This signifying which Dat is writable, if you have multiple copies on the same computer.

If you do not run the import command (which creates this for you), you should be able to create that file touch .dat/metadata.ogd manually and it'll work.

[rhodey]

@joehand thanks for the quick reply! I'm still running into trouble, my process looks like this:

1. HostA - $ mkdir /tmp/aaa && echo qqq >> /tmp/aaa/qqq
2. HostA - $ dat share /tmp/aaa
3. HostB - $ dat clone dat://<key> /tmp/aaa
4. HostB - $ dat sync /tmp/aaa
5. HostA - $ rm -r /tmp/aaa && dat clone dat://<key> /tmp/aaa
6. HostA - $ touch /tmp/aaa/.dat/metadata.ogd
7. HostA - $ echo zzz >> /tmp/aaa/qqq
8. HostA - $ dat sync /tmp/aaa

what I expect

dat on HostA notices that /tmp/aaa/qqq has new file contents (we appended zzz to it), HostA shares these changes with HostB, and the file /tmp/aaa/qqq on HostB is updated to contain qqq\nzzz.

what I observe

dat on HostA notices no changes to the file /tmp/aaa/qqq, using the flags --watch and --import has no effect, also dat on HostA doesn't seem to notice the creation of new files in /tmp/aaa either. the file /tmp/aaa/qqq is not "reverted" or anything to match the state of the archive, it is just simply ignored by dat.

it feels like dat still doesn't believe this archive is writable even with the touch trick. based on my interpretation of @tmcw's issue report I think we're running into the same problem, but if ya'll believe otherwise I'll be happy to open a separate issue report.

thanks for any help!

[joehand]

Can you try the same process but do dat keys export on the original folder and then dat keys import on the new HostA folder. That'll help to see if its an ownership issue or something else.

[rhodey]

@joehand I repeated the same process as before, but as instructed did a $ dat keys export of /tmp/aaa on HostA and then a $ dat keys import into /tmp/aaa on HostA after deleting and then restoring the archive from a peer. I was then able to modify the file /tmp/aaa on HostA and have my changes picked up and synced by the dats on both HostA and HostB.

So, that process works as I would expect :+1: it just seems that the $ touch trick might not be enough to make the archive writable by dat even while the ~/.dat/secret_keys directory is still in-place.

[jaccarmac]

@rhodey, when you did this, did you notice if syncing the Dat that was cloned as read-only then made writable manually caused it to index its files over again? Trying to transfer a writable Dat from machine to machine I'm running into a couple of issues: First is that the first sync causes the size of the Dat to immediately double as every file is added again. Second is that updates to the second writable Dat aren't reflected in the first.