nips

nips copied to clipboard

in most dynamic systems, let alone distributed systems, events that delete items from the state tend to be racy, very racy, and for users, losing their profile or relay lists or follow lists is pretty damn irritating

my suggestion is to change the filter in NIP-01 to add a flag that is default false, which when present enables the retrieval of a string of prior states of replaceable events

this doesn't change the logic of existing apps, but when the new filter field is seen, the relay can then instead send old states, and relays can then store old states and default to sending only the latest

this would be a big benefit to users enabling them to reassert prior states of their replaceable events

hell, it might make it possible to make most events replaceable and then make it so you can view the history if the relay stores it and if you ask for it

i will add this feature to my relay in the near future and it won't trigger unless the client knows it, this is an easy change that makes a big difference to user experience... add the flag, and keep the old events and not delete the old versions but retrieve "replaceable event" new versions by default

Not a bad idea.

But I wonder if it isn't better to just have "archival" relays that just return multiple versions of these events regardless of a different flag.

This doesn't add any new requirements in comparison to your idea since clients would already have to know what relays would be storing multiple versions of replaceable events anyway.

So, if nostr.wine, for example, wants to offer such feature, it could expose the archival relay under nostr.wine/archival or archival.nostr.wine.

You can also just request with a since that is lower than the current state, a bit more work for the client but requires absolutely no changes, the only difference is that relays that want to do this would just not delete replaced events.

These relays could announce this feature with NIP-66 to make the feature discoverable within the protocol

i have been thinking about it a bit more and after changing my profile and then exporting the database on it lo and behold yes it returns two versions

most request filters for this anyway specify they only want one copy, so what i'm doing today is changing the results code slightly so it just delivers them by default in reverse chronological order, newest first

it actually simplifies the implementation substantially, what started me thinking in this direction was removing the search and delete code and then last night it dawned on me, as i was watching people talking about losing their kind 0 event that "replaceable" should not in fact delete anything, deleting events should be a matter for garbage collection and cache management

the reason why i removed that code was because of the race condition i have been telling you about in eventstore/badger - it also unblocked the process temporarily but wasn't the solution because it still was racing sometimes - but now that the code is missing all i have to now do is add a datestamp sort to the events before it runs the limit filtering on it and this will also allow a higher limit value to return multiple versions - and it may also be necessary for client compatibility to default a missing limit in the filter to mean 1 so the behaviour follows the expected

You can also just request with a since that is lower than the current state, a bit more work for the client but requires absolutely no changes, the only difference is that relays that want to do this would just not delete replaced events.

These relays could announce this feature with NIP-66 to make the feature discoverable within the protocol

where is the NIP-66 draft?

You can also just request with a since that is lower than the current state, a bit more work for the client but requires absolutely no changes, the only difference is that relays that want to do this would just not delete replaced events. These relays could announce this feature with NIP-66 to make the feature discoverable within the protocol

where is the NIP-66 draft?

cc @dskvr

this is what me and @Semisol have been talking about for ages. it's a good idea. nostrdb already supports this since it doesn't actually "replace" events, it versions them.

Funny thing... Amethyst had to separate filters between replaceables and non-replaceables and manually add a limit:n for each replaceable we wanted to download as a protection to avoid receiving ALL past versions from non-compliant relays.

This is how we actually download the contact list after logging in:

Filter(
    kinds = listOf(ContactListEvent.KIND),
    authors = listOf(account.userProfile().pubkeyHex),
    limit = 1,
),

Most new relays or relays that were quickly built from scratch forget to make the special case of sending just the last event for the replaceable range + kind:0 and kind:3 and end up returning everything. This is particularly hard on replaceables that update very frequently, like Live Streams status events.

So, at this point, I am forced to already expect some relays to send past versions and always code accordingly.

Similar protections were also added for relays that are not compliant with ephemeral events (must add since: now()) and return thousands of stored NIP-46, NIP-47 calls.

In other words, compliance to since and limit are more common than compliance to specific behaviors per kind/kind range.

Similar protections were also added for relays that are not compliant with ephemeral events (must add since: now()) and return thousands of stored NIP-46, NIP-47 calls.

In other words, compliance to since and limit are more common than compliance to specific behaviors per kind/kind range.

In theory limit: 0 is supposed to prevent you have to use since: now(). In practice, some relays spit events at you anyway.

Versioned events is a good idea. It would solve #349. But it can't be relied on since the very high disk requirement would discourage people from doing it unconditionally.

On Sun, Feb 11, 2024 at 09:09:23AM -0800, Alex Gleason wrote:

Versioned events is a good idea. It would solve #349. But it can't be relied on since the very high disk requirement would discourage people from doing it unconditionally.

Occasional pruning of past events is a good idea in a full versioning setup. but information destruction as a default is the biggest issue with replaceable events. It's fine in many cases, but sometimes it sucks.

I could see keeping maybe the last 3 versions of each replacable event. But man those follow lists are brutal.

Similar protections were also added for relays that are not compliant with ephemeral events (must add since: now()) and return thousands of stored NIP-46, NIP-47 calls. In other words, compliance to since and limit are more common than compliance to specific behaviors per kind/kind range.

In theory limit: 0 is supposed to prevent you have to use since: now(). In practice, some relays spit events at you anyway.

what is "limit":0 supposed to give you? i set mine to interpret that to mean 1 if it's replaceable or parameterized replacable and max if it's anything else

I could see keeping maybe the last 3 versions of each replacable event. But man those follow lists are brutal.

they are, sometimes fills several screenfuls in my logs when they come through

can't really avoid the size on the wire but in the database you could be compressing these with a truncated hash and an index table

Versioned events is a good idea. It would solve #349. But it can't be relied on since the very high disk requirement would discourage people from doing it unconditionally.

events are inherently versioned by unique ID and timestamp, but different types of events probably should be pruned at some point... this is not a protocol issue though, it's an implementation question

On Sun, Feb 11, 2024 at 10:26:36AM -0800, Alex Gleason wrote:

I could see keeping maybe the last 3 versions of each replacable event. But man those follow lists are brutal.

yes which is why I've been so keen on a delta-encoded version for nostrdb 😅

what is "limit":0 supposed to give you?

limit:0 shouldn't return anything already stored/from the past. Only new events coming in live in the subscription should be sent to the client. Ephemeral connections use this a lot. If I send a "pay ln invoice" event, I want to observe the response to that event, which is going to be new, and not anything stored in the db.

what is "limit":0 supposed to give you?

limit:0 shouldn't return anything already stored/from the past. Only new events coming in live in the subscription should be sent to the client. Ephemeral connections use this a lot. If I send a "pay ln invoice" event, I want to observe the response to that event, which is going to be new, and not anything stored in the db.

what about if the field is missing, is this different again?

what about if the field is missing, is this different again?

Yep, then there is no limit. It should send everything you have that matches the filter.

I'm fine with this change, but I still think switching to better primitives for editing lists would be preferable. If we're keeping all updates anyway, the argument against add/remove events becomes irrelevant, since you would have the same number of events, but smaller. Compatibility, as always, is the problem with that.

I'm fine with this change, but I still think switching to better primitives for editing lists would be preferable. If we're keeping all updates anyway, the argument against add/remove events becomes irrelevant, since you would have the same number of events, but smaller. Compatibility, as always, is the problem with that.

Not really.

having to download and verify a bunch of events to compute the latest state is going to be more work than getting the state computed by the relay.

Relays can use deltas if they want to save storage, but there's huge benefits for a client to receive a single event with the state.

Many nostr devs work on long-running/frequently accessed clients, where the booting from no-state is rare, but if a new app has to download hundreds of events to boot that increases the friction for new players (although introduces the opportunity for minute-long spinners 👀👀👀)

Read optimizations can be supported by DVMs, race conditions having to do with writes can't be fixed without changing the event format.

Was thinking about this last night, there are two things that can be done to reduce the number of events that have to be fetched to get up to date: snapshots, and batch updates. For example, #875 does this:

Admins MAY publish member lists using `kind 27`. This MAY be published as a normal event, or wrapped
and sent to the group. An `op` tag indicates whether the listed pubkeys are being `add`ed to the group,
`remove`d from the group, or whether the member list is being `set`. An `a` tag MUST be included
pointing to the group definition event's address.

{
  "kind": 27,
  "content": "",
  "tags": [
    ["a", "35834:<admin pubkey>:<group name>"],
    ["op", "add"],
    ["p", "<pubkey 1>"],
    ["p", "<pubkey 2>"],
  ]
}

The reason I did this was to allow the group member list to scale past the maximum event size acceptable by relays, but it also allows an admin to set the list and add to it. The op isn't indexable, but a single letter tag could be used, which would make it possible for clients to 1. go back to the most recent set and 2. fetch all changes since that point.

I suppose only ops that are created AFTER the a's latest created_at count, right? So, to get the full list, you get the latest a + all kind:27s since the a's created_at and then apply them.

How do you know if the relay you are using has the latest a? It could be in another relay with it's own kind:27 changes.

Right, exactly. And because of the nature of nostr you don't know. But as long as you don't publish a new set you won't break anything. You'll be working off an inaccurate list, but you can still accurately publish updates to it. An optional prev tag could be used to indicate a missing set op, but I don't really know when you could be confident enough to publish it, when you wouldn't also be willing to create a new snapshot.

I described a very similar idea for collaboration to jeffg today where a bunch of pubkeys are p-tagged (ie. authorized) to modify a document and you only need to REQ since the latest replaceable event's created_at to compute the state and the original document's current state serves as a checkpoint/canonical (depending on the situation).

I think that's a cool idea, although I'm still concerned about how much it complicates the protocol if we use it for everything; isn't just the ability of easily rolling back to a previous kind:{0,3,etc) enough?

Delta events in basic kinds (0,3) certainly complicate things because we have to:

1. map out the complete event set to rebuild the state.
2. deal with multiple branches, with missing intermediary events, in separate relays/clients.

I don't think we should expect consistency where delta events are applied. If the lack of consistency is a dealbreaker, then we shouldn't use it.

Unbound lists are less consistent than the current replaceable event structure for lists but more consistent than the delta events (missing elements vs fully branching structures).

If you want to reconcile conflicting list updates you have to expand them into operations and rebuild anyhow. And in either case you've got to handle missing events. The two are really no different in terms of what you can infer in terms of eventual consistency. If we're worried about complexity, it seems to me unbounded lists won't help with that.

Maybe some back of the envelope math would help. Publishing all versions as a full list would result in a total of follows*follows/2 tags being published. Publishing a snapshot every 50 versions with add ops in between would result in follows*follows/100+(follows-follows/100) (or something like that)? So assuming a follow list that grew from 0 to 1000 pubkeys, in the first case you'd need to store 500k tags, but with ops you would need to store ~11k tags.

The two are really no different in terms of what you can infer in terms of eventual consistency.

I don't think so: Delta events need to deal with the missing events (same as unbound) but on top of that it has to deal with changes to a that don't have an op event, resetting ops. Or if the latest a itself is missing then it resets to the wrong state. If new ops are made in the wrong state and then the old a now comes back (the relay was unavailable for a few hours), things can get really messy. More messy than just missing events in an unbound list.

But yes, full lists are MUCH heavier to keep history.

Sorry, I think I led you astray. a doesn't have anything to do with what we're talking about, it's just the group's address. So there's no linking of events at all. But you're right that if a set is created without the correct preceding set event, you'll mangle the state. The point is if you're not 100% sure you should send set, don't, just send something that aligns with the user's intention (add/remove) instead. This results in self-healing the list because it gives the user the ability to authoritatively say whether a key is in the state at any time without involving any of the other entries. Sending the whole list every time means that if you're not in sync, the most recent entry may actually be wrong, and override a valid previous entry.

This is really just back to granularity; with kind 0's it's possible to update your website in one place and your lud16 elsewhere, and have the website get reset because the old value got set along with the lud16.

gone off on a tangent a bit here but just wanted to comment that deltas are a bad idea for nostr because of the lack of consistency

the thing that is missing from replaceable events is a tag saying replaces event <id>

then it becomes a blockchain btw

On Thu, Feb 15, 2024 at 03:57:27AM -0800, mleku wrote:

gone off on a tangent a bit here but just wanted to comment that deltas are a bad idea for nostr because of the lack of consistency

the thing that is missing from replaceable events is a tag saying "replaces event "

then it becomes a blockchain btw

I think there is some misunderstanding, The delta thing is a space saving optimization and a recovery tool, it is not a "fix" for the consistency issue.

I think a note ancestry spec would be interesting, depending on how important that is for your use case.