Add custom feeds

[staab]

This introduces a simple data format for defining custom feeds. It disambiguates between different feed types using keywords, so it's extensible. It extends the normal filter data format with some additional keys that can be interpreted using the current user's database to yield a standard filter. It also uses set operators to combine feeds in different ways.

The tricky thing about something like this is the balance between expressiveness and simplicity. For example, the union set operator is trivial to implement, but the symdiff operator can't be implemented without requesting all events for all feeds first. I've included all four major set operations in the first draft, but it might ultimately be best to rely instead on some external data source to do the heavy lifting for some of these.

For completeness, here are some justifications for novel additions:

• min/max WOT — I'm using numbers from 0 to 1 so that clients can define WOT score any way they choose.
• since/until ago — it would be useful to have a feed pointing to recent or less recent events.
• scopes — this is something Coracle has had for a long time. It allows applications to show different pubkeys to different users based on the user's own follow graph.
• DVM feeds — these are useful for defining any behavior that doesn't end up being possible using vanilla custom feeds.
• List feeds — useful for having a static pointer to a changing list.
• List of list feeds — I'm less sure about this one, but it could be useful for adding a layer of indirection when fetching lists. This could also be accomplished by a more flexible lists_by_filter type.
• Union feeds — for example "stuff from this list/dvm but also this pubkey"
• Intersection feeds — for example "stuff from this list/dvm, but with a minimum wot score"
• Difference feeds — for example "stuff from this list/dvm, except for this pubkey"
• Symmetric difference feeds — I've included them for completeness, but they're not very useful and much harder to implement.
• Relay feeds — useful for baking relay selections into any kind of feed in a way that's orthogonal to the other feed types.

One other thing to note is that some feed combinations can be compiled into simpler filters once lists are dereferenced, so the implementation is not as complex as it might seem at first glance.

You can see my initial implementation of this here. core.ts is the best place to start, since it defines all the data types.

[vitorpamplona]

Nice!

I would use this image from Niel Liesmons to illustrate the concept.

[staab]

I think most of what's in that mockup could be accommodated by this, minus mute words and display. But I think those are probably out of scope, and could be done client-side.

A few other considerations:

• Creating a UI that can handle these recursive structures is a huge pain.
• Since the format is extensible, people will extend it. Clients should have a mechanism for rejecting or repairing feeds that don't make sense.

[dtonon]

The combined feed are cool!

minus mute words

Couldn't a difference feed with a kind-10000 list be used?

[staab]

Couldn't a difference feed with a kind-10000 list be used?

You could, but since there's no way to compile mute words into a filter, that would have to happen in a post-processing step. Which is ok, but out of scope for me currently.

[fiatjaf]

Looks good to me, but since this is so big maybe we need a little more real-world experience that will maybe bring in some tweaks before we settle on a standard?

Is this a thing to be shared? Maybe sharing feeds is a good idea, because I can't imagine many users being able to fill a form such as the one in @vitorpamplona's image in order to construct personal custom feeds.

[staab]

My current use case is adding a custom tab to NIP 87 groups, which would involve adding a feed tag to the group's metadata. Because of the high level of complexity I'm inclined to make the first version of this NIP much simpler (basically just the filter feed type) but support everything in the implementation. As usage occurs we can add stuff back into the NIP. That said, I'm currently building a UI for editing feeds, so we might see that happen sooner rather than later, at least in Coracle.

[vitorpamplona]

Either this allows Clients to code their internal feeds in it and let people tweak a few parameters on an otherwise complicated algo OR it will require a FeedBuilder client, which I think will exist no matter what. It would be our own version of a decentralized Business Intelligence tool.

[staab]

Clients could also only support editing the simple filters feed, and only display other feed types, opening the feed editor for anything more complicated.

[staab]

Closed in favor of #1251