nips
nips copied to clipboard
nostr-protocol
NIP-69 - WIP - Source Control Context
I laid down a rough draft of how we can decentralize source control context using nostr. I am opening this Work In Progress PR so I can share it with others and gather feedback, as I'm certain that I missed something in my initial thoughts.
Are you working on this?
I was working on a git plugin myself to use git repos stored on Nostr, but eventually decided it was a bad idea. And it is better to use normal git servers for git -- and have them be self-hosted or hosted on a variety of service providers, then use Nostr to send open issues, send patches, comment on patches and for repository discovery.
I feel like these four functions above would make Nostr into an actually decentralized alternative to GitHub. Sending and applying patches was already done by @jb55 at http://git.jb55.com/git-nostr-tools, although it isn't being used much and I don't know if and how it could be improved.
The reason why hosting git data directly on Nostr is a bad idea is that it would be extremely inefficient and storage requirements would be massive -- so relays would start blocking this kind of thing and we would have to use special relays designed just for git-nostr, and at this point it is better to just use normal git servers that already exist. Each with its own specialty.
hosting git data directly on Nostr is a bad idea
Agreed. Also the git protocol is a very efficient optimized for git, and git itself is already fully distributed.
It's something I've dedicated some time to considering and working on a rough draft MVP implementation, but I wanted to open up the topic for discussion so I can hopefully learn from others that have given this some thought as well.
I 100% agree on not hosting git repositories on relays by default. But the discovery of git repos on a relay that decided to host the data could be part of the NIP, as I've outlined in the draft.
The NIP I began writing is outlining a way to store the context around source control. The draft here proposes using a tag in a specific kind to track the remote URL(s) for a given repository and creating an event from that. From there, clients can NIP-10 most things to create context like comments/issues/etc.
There will be metadata needed for tracking line-specific comments and that metadata could be included in the tags part of an Event of this kind.
QQ: How do NIPs get numbered?
@garyray-k I really like the idea of using nostr for context and keeping git hosting on git servers. I'm imagining a web-based nostr client that is essentially a github-inspired interface for working on PRs, issues, bugs, etc.
Having a conversation around a PR on nostr is easy. But I think we would need some kind of nostr-enabled git hook to publish events that represent code changes on the remote repo, like this:
The other thing is that making edits to a repository via this nostr-git-client seems like a potentially bad idea since putting private keys into nostr apps is still not ideal security-wise... NIP26 is out there now so maybe this is going to get better. But you could work on your repo from CLI or Git client as normal and see your pushed changes show up in nostr-git-client because they would be published to the PR by a hook on the remote repo.
Not having the ability to directly interact with a repo via the client would make things like forking more difficult and clunky though... No forking, no committing, no merging, no creating new repos...
QQ: How do NIPs get numbered?
I was also wondering this 😅
Having a conversation around a PR on nostr is easy. But I think we would need some kind of nostr-enabled git hook to publish events that represent code changes on the remote repo, like this:
I think the client would be responsible for this aspect finding new changes and updating the display. For an MVP, I envision the need for a client with two libraries, nostr and git (oversimplifying here). The git library is largely reading repositories and the nostr library publishes events with metadata to tie the event to a specific part of the git repo.
I've been experimenting with this for a couple of days, and made decent progress, hit a few brick walls here and there but I thought I would give my input/conclusion.
I started with a script that I could use to publish git refs (heads/tags) and objects (commits/trees/blobs) to a unique repository, the plan here was to get the event structures/schemas sorted before porting this script to a git-remote-helper so that we could simple push/fetch using git with a custom remote handler
git remote add origin nostr://some.relay.io/814add12473702fcf229d4786bc46e5ddbabbb88027037e371ec4c7947de284d
After getting this far, I felt quite confident this could work, so I set about creating a client to interact with these events, that turned out pretty straight forward, I was able to create new repositories, update their details/settings etc... once a repository was created I could push branches up (with my dev script / not a git-remote-helper yet). At this point, I could browse all of my repositories and see all of their refs along with view the tree and latest commit of each ref.
What I found was, all refs/objects are quite light minus blobs, moving over to packed git objects (deltas) rather than loose objects (no deltas) would certainly help here but then I thought about maybe storing blobs on the filesystem outside of the database (but this would rely on relay changes and I was trying to do this without introducing any new relay features than already proposed). Turned out it wasn't actually the end of the world with the files I tested on since my UI was only ever subscribing to blob objects when viewing a file.
Few screenshots of the progress
So things where looking good, however this was all just a big experiment, in the end I 100% came to the same conclusions that have been mentioned. Although I found there wasn't any performance issues with the test repos/branches I had published to a relay I'm sure with larger repositories it could be a concern.
The funny thing is, before I started any development I wrote about a few options, having a git-server was the in between option
All that said, it was a fun experience, I learnt a tonne of git internals along the way so it wasn't a complete waste of time but I think it might be time I hang up the boots on this one. If anyone want's to discuss how a git-server model may look, I would be up for discussing / working on something.
QQ: How do NIPs get numbered?
What I've done is look through all the PRs, find the highest number, and go one higher.
I like this draft and the initiative. Thank you @garyray-k.
I agree with the idea of having a dedicated kind for "merge requests" (and possibly one for "issues"?), but the responses to these seem to me to fit better in kind:1 events.
It would be nice to have these notes -- i.e. discussions about a merge request -- to show up in the normal "social" feed of users, right? And then social feed users could click on the context to see the event they were replying to and then what would they see? An event (that they would not understand) specifying a merge request. Do you think this is good or bad? Did you think about that?
It seems we are all here in agreement that relays should be a transport layer rather than a storage medium. This points to git servers holding the data and communicating over Nostr with git clients. In my opinion, censorship resistance is very important for open-source code moving forward, and Git on Nostr can supply it. The trick is having publicly known code repository entities to which public communication is connected while having their data "hidden," private but verifiable. I think the solution, in short, is that Nostr-authenticated developers can use a Nostr-compatible git client to push to and pull from Nostr-authenticated git suppliers using Nostr relays as a communication layer.
It can connect with @gayray-k suggestion above. I am not sure I have the time to create this on my own. I would appreciate any remarks on this and help get more people on board with the idea. For a more elaborate explanation, please see this google document: docs.google.com/document/d/1z96XpTtrUvrFZYPjMzlT5JuE41HF6X5vI99lvImVCvU
@hendore Nice research! I'm glad you concluded that relays mustn't hold repos, but there will likely be relay servers that do both for hosting simplicity.
It would be nice to have these notes -- i.e. discussions about a merge request -- to show up in the normal "social" feed of users, right?
I think this will happen with kind: 1 events as replies. To me, it makes a relay sort of a town square. You might not care to hear about merge request discussions, but by being on the "global" feed of a relay you will see it nonetheless.
And then social feed users could click on the context to see the event they were replying to and then what would they see? An event (that they would not understand) specifying a merge request. Do you think this is good or bad? Did you think about that?
I will have to do some research on how clients are handling the "content" field. My initial thought would be to standardize the content field of the merge request event so a client that does not explicitly handle "merge request events" can still display something meaningful. Along with this research, I wonder if clients filter based on root events. That seems possible but it might not scale to a certain point.
- [ ] research how some clients handle unknown
kinds
I think the solution, in short, is that Nostr-authenticated developers can use a Nostr-compatible git client to push to and pull from Nostr-authenticated git suppliers using Nostr relays as a communication layer.
Git already handles authentication and even has mechanisms for separate authors and committers. The goal of this NIP would be to anchor series of events to a single git "thing". The simplest I reasoned about was a merge request. How would Nostr auth enhance existing git authentication?
My initial thought would be to standardize the content field of the merge request event so a client that does not explicitly handle "merge request events" can still display something meaningful.
That's kind of what I was thinking.
Git already handles authentication and even has mechanisms for separate authors and committers. The goal of this NIP would be to anchor series of events to a single git "thing". The simplest I reasoned about was a merge request. How would Nostr auth enhance existing git authentication?
My purpose with "Nostr Auth" is to separate the constant public point of contact from a dynamic, nonconstant uncensorable IP generated for a git session. At the first simple stage, this IP may be the IP of a git server that is generated by a VPN per connection request. At later stages, the entire client-server git communication can be done by both the client and the server polling on private messages sent to them and transmitting data over relays with no server IP involved. This does not help in git authentication. It helps enable a censorship-resistant public git service that can synchronize open-source development.
@ofer-elrom While I appreciate the consideration, I think what you're describing would be outside the scope of a NIP. From my understanding, you're trying to create a new way for git to be communicated/networked. That would not be the goal of this NIP. This NIP is aimed at merely defining a way to reference source control primitives (ie. merge request) as a nostr event so further context can be built from the root reference point.
While I appreciate the consideration, I think what you're describing would be outside the scope of a NIP. From my understanding, you're trying to create a new way for git to be communicated/networked. That would not be the goal of this NIP. This NIP is aimed at merely defining a way to reference source control primitives (ie. merge request) as a nostr event so further context can be built from the root reference point.
Thank you. I understand. Sorry for the distraction of focus.
I like were this is going. Feature-parity with GitHub reviews (e.g. resolving inline review comments) is one of the evaluation schemes used when thinking about GitHub alternatives for Bitcoin Core (src) (and I assume for any other large-scale open source project).
Maybe https://github.com/zw/bitcoin-gh-meta can give some inspiration on the required context schemes.
Someone brought up a good point of consideration: how do disparate notes maintain their tie together?
Example: Dev A creates PR and the root event for the source control context. Dev B creates a reply note using NIP-10 Dev A and B are publishing to different relays. Dev B's relay goes dark and nukes all data.
How do we maintain this context?
Nostr provides redundancy if you utilize it. Dev B should post to more than one relay.
Now the trick here is that people following the repo need to see Dev B's note, so instead of what we do for a microblog (everyone writing wherever they prefer), repos should probably be hosted on a (per-repo) well defined set of relays (but more than just one) chosen by the owner of the repo, and these relays configured in a way that they allow access appropriately to this repo.
The NIP-65 model surely won't work. It's not 1-to-many, it's many-to-many.
my proposal for Git:
- ✅ store on Nostr
- issues and patches, comments to them, their state, labels, assignees, ...
- information about the current refs (branches, etc)
- repository information, name, description, etc
- ❌ store externally
- the actual objects (discover git repos that may contain this data through nostr)
- the patch contents (may be too large)
- file attachments
The one assumption github makes about the git repository itself is there is only one official location to push and pull from. This makes sense in the case of something that's intended to be a frontend to a git server, but it doesen't leverage the decentralized intent behind git.
Github stores PRs on the central repo on a branch, which requires write access to the central repo. It makes more sense in decentraland to point to a branch on a git remote rather than requiring the project owner to manage a ratsnest of ACLs for everyone that wants to push.
Github solves this by allowing references to forks inside the platform, but this is clunky and clutters your own account and if you dont intend to actually fork the project then you will never make use of all of the other features in your own account.
It would be nice to have a "pure" github URI place to store your public copy of the repo without needing to access it through a bloated CMS like github. This could be as simple as your own ssh server and then anonymous read-only access is provided to the "Project context" as (multiple?) URI and sshkey fields. Alternatively, there might be future integration with git-torrent to host the central repo.
In the case of building it on a nostr backend, each PR should point to a unique http/ssh uri (which does not preclude using the central repo).
FYI: NIP-17 might be related and of some interest
https://nip17.com/
I've been thinking about this for a couple of years since before nostr, and I realized that nostr pub/sub was the missing piece I needed to have a source control system without vendor lock in
I've decided that full source control or full github is too much to take on and achieve, so I will focus first on just commits, which can lead to continuous deployment/integration without lock in, and also ability to move and discover repositories. After that is working hopefully extensions. Will follow this work with interest!
@kirkawolff
Github stores PRs on the central repo on a branch, which requires write access to the central repo. It makes more sense in decentraland to point to a branch on a git remote rather than requiring the project owner to manage a ratsnest of ACLs for everyone that wants to push.
The owner of the root repository Event would be listing the remote locations that are canonical. How they manage ACL for those is outside the scope of this NIP.
It would be nice to have a "pure" github URI place to store your public copy of the repo without needing to access it through a bloated CMS like github. This could be as simple as your own ssh server and then anonymous read-only access is provided to the "Project context" as (multiple?) URI and sshkey fields. Alternatively, there might be future integration with git-torrent to host the central repo.
This already exists. You can host remote repos outside of github that people can clone from. This is a decent article describing the process (https://www.inmotionhosting.com/support/website/git/git-server/). And if you look at a GitHub URL, it's a remote URL + file path.
Someone brought up a good point of consideration: how do disparate notes maintain their tie together?
Example: Dev A creates PR and the root event for the source control context. Dev B creates a reply note using NIP-10 Dev A and B are publishing to different relays. Dev B's relay goes dark and nukes all data.
How do we maintain this context?
This is IMHO quite a hard problem so solve fully. There may be some partial solutions, but git trees and commits can get quite complex. There are also race conditions, which ultimately may need to be byzantine fault tolerant.
Hence, I went for a simpler approach, and that is that the repo will be able to publish the definitive state to a relay. Future extensions would deal with the different race conditions. This is similar to continuous deployment type models. Once that is working, more could be attempted.
How do we feel about adding the POSSIBILITY to include the patch in a merge request event somehow?
It would enable users to create merge requests without having write access to any git server. Could be a confusing first time experience if you need to find a git server to host your changes. Clients would probably be forced to have default git servers for their users just like with images now.
The obvious downside is that including the patch would create large events. Completely agree with the general sentiment that the relays should not store the repository. Sending patch files around is the OG way to use git so it could make sense to support them in this brave new word. Maybe another kind and another NIP?
For example this is the patch for this PR https://patch-diff.githubusercontent.com/raw/nostr-protocol/nips/pull/223.patch
Posting note1qj84ua9mklntsqvk2khytfpwj0jtq297x84u7k4de6nq9rz060kq9zt84j here for context:
#[3]
I think the way this bounty is stated is put is not the ideal. I think most people will read this and think we need a big website that is just like GitHub but using Nostr somehow. I think that is not what we should see (and hopefully that's not what #[1] wants either).
What I would want to see are multiple apps that can interoperate and are able to perform separate functions:
- browse code
- comment on code (referenced by a commit)
- create issues and comment on issues
- send patches
- comment on patches
And how these should be done? I am not sure, but here's what I have in mind:
- most of the comment things should probably be kind:1 events, I don't know, with some extra tags (so they could be interacted with from the normal "social" Nostr clients? or not?)
- code should probably be hosted by standalone dedicated git servers -- and there could be centralized providers offering these services but they should interoperate seamlessly between themselves and with standalone personal servers
- sending patches should probably be done using something like this approach by #[0]: http://git.jb55.com/git-nostr-tools/file/README.txt.html
#[2] has opened a discussion on this topic on the NIPs repository that could possibly be used to coordinate the efforts: https://github.com/nostr-protocol/nips/pull/223
I think we could have multiple different smallish webapps, native apps and specially command-line tools that implement one or multiple of the separate functions described above, and with that we can achieve a much better result both in terms of quality and of decentralization than if someone or some big team decides to tackle the entire cake and come up with some centralizing architecture on their own.
@melvincarvalho
This is IMHO quite a hard problem so solve fully. There may be some partial solutions, but git trees and commits can get quite complex. There are also race conditions, which ultimately may need to be byzantine fault tolerant.
Hence, I went for a simpler approach, and that is that the repo will be able to publish the definitive state to a relay. Future extensions would deal with the different race conditions. This is similar to continuous deployment type models. Once that is working, more could be attempted.
By publishing the definitive state of the repo to a relay, you're recreating git. That is not the goal of this NIP. Tracking which git remote repository is the definitive state is encompassed here using the b tag mentioned in kind: 69 events.
What about using IPFS at least for public repositories ? For example automatically commit a ".bundle" file to IPFS to make it accessible even if the origin git server has been shutdown in some way....
Commenting note1tgy2myzq7g5dyckjm47rm7qsmu48dktgna7m5smcp4qng4kgha8qa5n56g for context.
Linus wrote git to be distributed (decentralized) and his workflow with email patches was a core design feature.
Github projected a centralized workflow onto git (branch permissions and all that jazz are not a part of git). This has its place especially in the enterprise world. So why do we want branch permissions for open source projects?
I think it comes down to trusted reviewers. There are those rare combinations of talent, common sense and OCD that make excellent master branch guardians.
So all I think we need is
- the feature of broadcasting merge requests on nostr
- the feature of trusted reviewers signing their approval to this request
- the feature of cryptographically coming up with a deterministic merge outcome and commit hash so every distributed client would have an identical merge
- a git client/client extension to automatically merge based on preconfigured sign events
Every developer could still hold the code local, plus there could be X public repository servers.
How do we feel about adding the POSSIBILITY to include the patch in a merge request event somehow?
It would enable users to create merge requests without having write access to any git server. Could be a confusing first time experience if you need to find a git server to host your changes. Clients would probably be forced to have default git servers for their users just like with images now.
The obvious downside is that including the patch would create large events. Completely agree with the general sentiment that the relays should not store the repository. Sending patch files around is the OG way to use git so it could make sense to support them in this brave new word. Maybe another kind and another NIP?
🤔 I don't see a reason not to aside from size of the event, but perhaps larger patches could be serialized into a thread of events. I would also imagine this could promote small patch sizes.
The critical thing touched on here is the barrier to entry for a new contributor. Spinning up a remote repository is not trivial.
Edit: Thinking more on this, and it may make sense for this to be its own kind of event. That way the content of the event is just the patch text. Then clients can easily filter out those kinds where applicable.
FYI: we decided previously on telegram to use the tag "commit" for a git commit
https://t.me/nostr_protocol/76815
This is possibly what you are calling a "merge"
@MazziaRick
What about using IPFS at least for public repositories ? For example automatically commit a ".bundle" file to IPFS to make it accessible even if the origin git server has been shutdown in some way....
This is a git implementation detail and outside the scope of this NIP. IPFS or otherwise, the remote accessibility of a repo would be in the b tag if adhering to this NIP.