nostr-protocol
NIP-34 - Algorithmic Filter
Before going on with NIP text, I would like feedback on the concept below. It is simple but hard to explain. It is a way to enable algo decentralization by making algos work in any dumb relay instead of Google algo that works only on Google relays. Please tell me your questions if any.
From NIP-01: When limit: n is present it is assumed that the events returned in the initial query will be the latest n events.
With the above quote in mind, we could turn this:
["REQ", <sub_id>, { kinds: [1], ..., limit: 5 }]
Into this:
["REQ", <sub_id>, { kinds: [1], ..., limit: 5, nip34: 'algorithm_a' }]
Then, non-supporting relays will (probably) just ignore the nip34 field and sort by created_at in descending order, as usual (this detail is important because it is what makes it backward-compatible).
While supporting relays (that not only support nip34 but also algorithm_a) will just swap the created_at field in their internal query with a nip34_algorithm_a field.
I want it to be a simple swap of fields. For example, if a relay implementation uses a SQL DB, the above filter would turn this:
SELECT * FROM events WHERE kind in (1) ORDER BY created_at DESC LIMIT 5
Into this:
SELECT * FROM events WHERE kind in (1) ORDER BY nip34_algorithm_a DESC LIMIT 5
So:
- Each NIP-34 algo supported by the relay will in practice add a field on relay DB for each event. The field works like created_at, meaning the event with highest value in said field will be the first retrieved on queries;
- Each specific algo section explains how and when to update the corresponding field. For example, the field is always set on event creation but may be updated every time the event was "liked"
If the concept is acknowledged, I will kick start the NIP with a simple algo that will enable us to sort events in ascending order. Such feature is useful for instance on github-like PR discussions that have older messages at top.
Should each algorithm be a new nip like NIP-3401 (instead of just "algorithm_name" inside a NIP-34 section) so that relays can advertise support on Relay Information Document using "supported_nips" array field?
I see there's also a "supported_nip_extensions": ["11a"] field example, so maybe "supported_nip_extensions": ["34algorithm_name"] or just "supported_nip_extensions": ["34a"]?
Joy!
Would it be extra to make nip34 take an array? Some algorithms might take arguments that clients might want to expose. It could be best to keep it simple though and add some other nipXX field later for args.
@huumn What kind of args you anticipate? This NIP seems incompatible with args cause the score field isn't calculated on each client query but previously set on event creation but may be updated every time the event was "liked" etc. So it is not a different score for each user.
However, the good thing of being a REQ extension is that the client can be smart by keeping track of pubkeys from whom the user is consuming content, also hashtags, and use such pubkeys/hashtags as a way to tailor the query to that specific user like this:
["REQ", <sub_id>, { kinds: [1], limit: 5, nip34: 'featured', authors: ["p1 user follows", "p2 user has read content recently", "p3 user has liked content recently"] }, { kinds: [1], limit: 5, nip34: 'featured', '#t': ['cats', 'bitcoin'] }]
This NIP seems incompatible with args cause the score field isn't calculated on each client query but previously set on event creation but may be updated every time the event was "liked" etc.
Sorry I missed that. Should we be telling relays how they should be ~~computing~~ keeping scores? It makes sense to suggest these things though I guess.
So it is not a different score for each user.
In the case of ranking by likes I guess this would mean my feed could be influenced by anyone, so long as other filter conditions are met, ie there's no way for me to filter out the influence of likes by people I've muted for instance.
I do really enjoy how simple this is. Nice work.
What kind of args you anticipate?
One example is whose likes we want to take into account. The simplest version is just a boolean flag, e.g. true if I only want my follows considered, false for global.
Each algo section on the NIP describes how relays should compute the score. It's a global score so can't avoid muted ppl weight. Relays should atleast rate limit likes and such by ip to avoid gaming the score.
This seems really similar to the atproto solution. That's one of the few areas where they're farther along than nostr.
https://github.com/bluesky-social/feed-generator
Took the route of adding NIP extensions for each algo to use "supported_nip_extensions": ["34a", "34b", ...] etc from NIP-11. Added NIP-34a Oldest First algo.
This sounds like a good solution, but before this is tried I think we need a more developed client ecosystem first, in which clients are more prepared to differentiate between relays.
On the other hand, if clients get to that point, it would be easier for a relay to just offer different endpoints which users can specify manually by using URL paths. This would ensure maximum compatibility with clients that do not explicitly support algorithms.
it would be easier for a client to just offer different endpoints which users can specify manually by using URL paths.
The client? Or you meant a relay besides supporting { "nip34": "algox" } filter field could make available wss://relay.url/nip34algox in addition to regular wss://relay.url for users to add to relay list on unsupporting clients?
@fiatjaf I've added it as a requirement.
Instead of a new endpoint, ended up using a ?nip34=algo query param on connection string cause there is already things like wss://filter.nostr.wine/npub1...?broadcast=true in the wild and I think requiring query param support is less complicated than appending /nip34a to the path like wss://filter.nostr.wine/npub1.../nip34a?broadcast=true .
It ocurred to me that the algos don't need to return a string because DBs that can only index strings already have to deal with integer for the created_at field. So they store string but turn it into integer before replying to the request, or index the string but store both versions. I will edit it.
Wherever we're heading with this, I think the entire thing should be in a single file.
I'm biased but if this gets merged and relays start supporting this, the protocol gets both a simple way to do client-relay event sync (seen_at) and to reverse sort events (asc). Huge win for clients that keep a local DB so would appreciate sync or to those that don't have a local DB and would like to reverse sort events relay-side.
More involved algorithms can be added in the future as an extension. Like NIP-11 "supported_nip_extensions":["34-extra-algo-xyz"]. Regular NIP-34 support would be achieved just by implementing seen_at and asc.
Why do this with an extra parameter in the filter instead of a subdomain or a path in the relay URL? I think the later is better, more flexible, more transparent and easier to grasp.
i.e. instead of sending ["REQ", "_", {"nip34": "crazy"}] to wss://relay.com you could send ["REQ", "_", {}] to wss://crazy.relay.com or wss://relay.com/crazy then it is immediately obvious to anyone what is happening and no one has to code new parsers or anything like that. Also it works immediately on all clients that allow users to select relays to query from, they can just paste a new URL there.
I'm biased but if this gets merged and relays start supporting this...
Not to be annoying, but the order is: relays start supporting it and then it gets merged. :)
@fiatjaf because the same connection can be kept open and reused for subscriptions that use or don't use NIP-34 (side note: the field name is now "algo" instead of "nip34" cause don't know if gonna have to change nip number).
I already added the Relay Connection URL Query Parameter section with what you said, to be able to manually add relay urls on clients that don't support this nip.
@vitorpamplona fair T_T I haven't seem many relay implementers on NIP discussions lately