dat-ecosystem
Further integrating Beaker and the dat cli
One of the challenges we have in Beaker is giving developers good command-line tooling.
Ideally, users would be able to Beaker's development and publishing behaviors from the command-line. We explored this briefly with bkr but deprecated it because we disliked the results.
One top issue with bkr was that it provided many of the same functions as the dat cli, which we felt was overly confusing. It forced users to ask, "When should I use bkr vs dat?" and "Why do I need bkr at all?" So, I believe the community will thank us if we can provide most of the value in the dat CLI.
Our goal is to make dat and Beaker work well together if possible. Ideally the two could be used interchangeably; an archive created from dat should be writable from Beaker when both are running on the same device.
Integrating dat and Beaker presents new problems. The current dat-node backend is designed for single-thread access and does not expect that any other processes will write to the SLEEP files. There are many possible constraint violations when multiple processes are involved; at the file level, blocks could get overwritten; at the data-structure level, the hypercore logs need to maintain a monotonic sequence. Therefore, coordination is required.
Coordination could be accomplished over network sockets between the processes. This has the advantage of being usable with remote devices as well. However, as I explored this option, I felt it was not ideal to represent Beaker as a remote endpoint. Interacting with remotes can already be complex, and overloading the concept with local processes seems like it invites confusion.
Is it possible to use use file locks to create transactional writes to the SLEEP files? If so, then we can enforce the data model's constraints with very little configuration by the user. Note: I assume portable file-locking has been solved in node, but I haven't researched this fully.
cc @mafintosh
Why not get rid of dat-node and directly connect to dat itself if installed on platform? Case not installed, Beaker haves it built-in and does the work to install it, then connect to. Also, you've blocked me on Beaker's repo (same behavior the Dat developers, that are serious and respectful, didn't did same)?
cc @pfrazee
We're finally (hooray!) creating a shared daemon for all dat tooling. It should deploy this year.
@da2x I've unblocked you. I don't recall why you were blocked and we should've explained why when it happened.
Why not get rid of dat-node and directly connect to dat itself if installed on platform?
I don't mean getting rid of dat-node at all, but on Beaker browser; I understand the point of dat-node as a module for independent apps that wants to run natively instead of on the browser.