Questioning JSON peculiarities

[burdiyan]

This was initially part of https://github.com/nostr-protocol/nostr/issues/127, but I decided to separate it as these more subjective concerns of mine.

I don’t know if I’m the only one here that thinks this, but I found many things in Nostr's requirements about JSON to be peculiar and unusual. I also couldn’t find any justifications for these choices in the NIPs, so maybe it would be worth adding something to make it clearer.

Tags

Why event tags aren’t just fields on the JSON instead of being an array of arrays? The most confusing part for me was to find that inside the subscription query request, they are indeed fields on the JSON prefixed with #. Why couldn’t they be the same in the event structure?

Something like:

{
    "id": "...",
    "kind": 1,
    "content": "...",
    "#p": [[<public key>, ...]], // Or object instead of array.
    "#e": [<event-id>, ...],
    ...
}

The fact that tags are “tuples” (in fact they are like function calls with some expected arity) makes them much harder to work with and to evolve, IMO. E.g. the p tag can have 2 or 3 “arguments” (with or without petname). I think this would be so much easier to extend and work with if they were JSON objects instead of tuples. In typed languages like Go, people normally prefer to work with structs rather than tuples. So converting these arrays into structs would require custom marshaling/unmarshaling code, which is always cumbersome. Having these as optional top-level fields in the event's JSON object would be so much explicit and easier to work with, IMO.

Structured Content

The most surprising and confusing part for me was to find out that content is strictly defined as string. So, when working with structured content you’d have to encode JSON as a string and put it into something that’s already a JSON, which feels a bit weird.

Again, I’m sure there’re reasons for these choices, but I couldn’t find any in the NIPs.

[fiatjaf]

I agree the choices feel odd, but they were made such that the event structure was predictable and static and that hashing the event was simple and yielded a deterministic result.

The # in filters exists to denote that these keys represent tags.

Also for some weird reason these "tuples" or "function calls" appeal to me very much, because there is no doubt about the name of the property, you can't type the wrong name and get away with it, you don't have to memorize the names, and also you don't have to transfer the names around.

[burdiyan]

I agree the choices feel odd, but they were made such that the event structure was predictable and static and that hashing the event was simple and yielded a deterministic result.

I assumed that the reasoning was related to hashing and signing. Maybe worth mentioning this in the corresponding NIPs?

On the other hand though, I believe there could be much simpler ways to go around this. Without talking about IPLD which could help a great deal with having a canonical encoding, one approach for signing JSON that I find really-really elegant and simple is what is implemented in Perkeep. Really, do take a look. It's just soooo simple, and elegant, and doesn't limit one to have any JSON structure at all, with nesting and etc. If I wouldn't care about binary encodings, and only cared about JSON, I would implement the signing exactly like this, without any doubt (ignoring the GPG stuff 😀)!

The # in filters exists to denote that these keys represent tags.

I did get that :) What I didn't get is why tags can't be stored as fields in the event object as keys in the similar way. I get the point of determinism, but in addition to other solutions I mentioned for signing, one could simply sort object keys and throw the values into the array form to be hashed and signed.

Also for some weird reason these "tuples" or "function calls" appeal to me very much, because there is no doubt about the name of the property, you can't type the wrong name and get away with it, you don't have to memorize the names, and also you don't have to transfer the names around.

I didn't get this part, sorry. How is there's no doubt about the name of the property? You have to lookup the spec to know what tags are allowed, and it's not conveyed by the code you're looking at (which would happen if these were properties on a struct). How's that you can't type the wrong name and get away with it, if basically "anything works" inside the tags field?

Also, how's that you "don't have to memorize the names"? And how is that you "don't have to transfer the names around"? Could you maybe expand on that?

[mikedilger]

String content and array order don't change based on different JSON serialization implementations, so signatures will match. JSON objects with keys may come out in a different order and screw up the signature verification.

Even if we were all convinced there are better choices, it would be quite a thing to change now, given how widely nostr is currently deployed.

The way I personally dealt with the issues you are raising is to build a library nostr-types which strongly types tags and gives them field names.

[burdiyan]

String content and array order don't change based on different JSON serialization implementations, so signatures will match. JSON objects with keys may come out in a different order and screw up the signature verification.

I get this! What I'm suggesting is to add a note in the corresponding NIPs to explain the reasoning.

Even if we were all convinced there are better choices, it would be quite a thing to change now, given how widely nostr is currently deployed.

If there's willingness to change some of the fundamentals, I guess there will be no better time to do this, because there will only be more users and more clients as time goes by.

[mikedilger]

I get this! What I'm suggesting is to add a note in the corresponding NIPs to explain the reasoning.

I support that.

[sant0s12]

@burdiyan I was curious about this too so I'm happy to have found this thread :)

I think that the signing strategy that Perkeep uses is very nice indeed and avoids the undefined serialisation order.

I wonder if there is any chance to port this to nostr since it would make working with tags a lot nicer but I'm not optimistic.