nostrability
nostrability copied to clipboard
nostrability
end user cannot read from relay (relays reject large contact lists )
client initiating action
nostrudel
client receiving or observing action (or lack thereof)
nostrudel
relays
relay.mostr.pub nostr.mom momostr.pink nostr.bitcoiner.social nostr.fmt.wiz.biz relay.nostr.band nostr.bitcoiner.social
timestamp
https://njump.me/nevent1qqs2z0f52dn67j3xc79cpxhzapdfhct95dmluhn84rz5zs28fww3z5spz4mhxue69uhhyetvv9ujumt0wd68ytnsw43qzyrhwden5te0dehhxarj9emkjmn9qy8hwumn8ghj7mn0wd68ytnddaksz9rhwden5te0wfjkccte9ejxzmt4wvhxjmcmsjtzk
reported by @0xtrr
what happens
invalid: event too large: 69764
with 947 follows
suggestion
How might relays deal with large contact lists?
@0xtrr suggests relays can expose this limitation via NIP-11 implementation
https://github.com/nostr-protocol/nips/blob/master/11.md#server-limitations
You can find this document by sending a HTTP request to the relay url with the "Accept" header set to "application/nostr+json". My relay uses strfry which doesn't expose that information unfortunately. You can test it by sending this curl request:
curl 'https://nostr.oxtr.dev' --header 'Accept: application/nostr+json'
See also: https://github.com/nostr-protocol/nips/issues/1179
I am very passionate about this issue. We just need to convince @fiatjaf to bend the fabric of space and time.
Suggestion: delta addition by @water783
Https://njump.me/note1hagrttg5g3urdadu6m8y0wg7emps5wz2yu46ewdrwclyeu08fadqnrm88a
The relay should no longer dump. We need to add a “add to follow” event, and then the relay should automatically add the user to the follow list, instead of updating the entire list at once.
This has been discussed a million times, the consensus is that some people think this is the worst problem to ever have happened to humanity while others think it's not a big deal and that the cure may end up being worse than the disease.
Why do you have to store your huge contact list in all these relays anyway?
To be honest contact lists should have never existed anyway -- or relays should not index individual tags in them.
@fiatjaf This isn't even about indexing. This is just the event being so huge the relay rejects it.
I suggest that the relay can supports downloading large events via HTTP/HTTPS, with resumable downloads, instead of using ws/wss requests.
@vitorpamplona mentions the issue is greater than contact lists
Anything that can have a bigger blob, like large blog posts, HTML pages (from hostr.cc) documents, encrypted spreadsheets, private lists, long DMs, large zap split setups, group membership events, etc.
We will get into a point that most events will simply be rejected. So yeah, we need to figure out a more accurate way for relays to expose what they support. NIP-11 is nowhere near enough.
This is just the event being so huge the relay rejects it.
@alexgleason
For some reason relays don't want to store a bunch of data for free for anyone. This is perfectly normal and expected.
If you break the contact list in multiple events they will take more space in the end. That may bypass the current naïve limits these relays are imposing, but ultimately, if relays really want to save disk space, they will have to block these too, and much more aggressively.
Now, in reality, relays are probably blocking these contact lists because they don't know these are real contact lists, they are not aware that contact lists can be huge, they're just seeing a big event and blocking it thinking it's some base64 video file.
The solution here is not to try to be sneaky and bypass current limits creating a potentially worse future for everybody, the solution is to improve technology so relays can properly handle contact lists that they want to store while also protecting themselves from base64 videos.
I mentioned indexing because that's another reason relays might not want to store these lists, because they have a lot of tags. And that's another thing that should be fixed by improving relay codebases.
Honestly, with a dozens lines of code this wouldn't be a problem.
With that said, I like the idea of multiple events myself at a conceptual level, it is certainly more "correct" and more elegant. I'm worried about its ultimate practicality gains, but I think it should be tried.
Honestly, with a dozens lines of code this wouldn't be a problem.
To what? Relays, e.g. strfry?
To what? Relays, e.g. strfry?
Yes.
Since strfry guy is gone, does it make sense to post a bounty for these changes?
He is not gone, and I don't think it makes sense to ever post bounties anymore. Also we don't know exactly what would be these changes. What I was trying to say is that relays need space to make more fine-grained decisions about what they're going to store, but the exact best interface for that is up for experimentation. You could go very far using strfry plugins though, which are very customizable and already in use by many relays out there.
or relays should not index individual tags in them.
Would make counting followers a bit more laborious.
Note: Fat-fingered a hotkey to close the issue, whoops.
I don't see the point of publishing large follow lists in public and I hope relays will keep rejecting them.
I don't see the point of publishing large follow lists in public and I hope relays will keep rejecting them.
What solution do you propose?
I don't see the point of publishing large follow lists in public and I hope relays will keep rejecting them.
What solution do you propose?
On the client:
- somehow encourage to clean up follow list
- auto clean up? offer the remove inactive accounts
In general:
- if public, it should be useful for others
- if private, then it should be encrypted and maybe even on a private or paid relay so you can make it as big as you want
- find out the cause, how or why did the follow list get so big? maybe it shouldn't be a follow list but a different kind of list
I don't see the point of publishing large follow lists in public and I hope relays will keep rejecting them.
really? it's pretty common on twitter for accounts with 10k+ follows. people have the expectation that they can follow as many people as they want. We can say "too bad", but it feels like we are just giving up and working around a bad design to begin with.
I like the solution of replaceable events for updating a follow status for a given pubkey. then you can sync all of these statuses incrementally.
The only problem after that is reducing the subscription size for large number of follows. outbox helps this a lot, but in the degenerative case where a large number of people share the same relay it might make sense to make an optimization that simply does: ["REQ","timeline",{"timeline": "pubkey..."}], but this is an ugly custom hack for a particular usecase. in nostrdb I am working on wasm queries: ["REQ","timeline",{"script": "nscript..."}] which could build a timeline query from a contact list on the relay, but I doubt this will be adopted in general, maybe only for AUTH paid relays.
In damus some of our users were experiencing stalling with rejected queries that were too large, we split our timeline query into batches, but this feels like a hack to get around relay query size limits. Batching may be the only realistic solution that works today.
if private, then it should be encrypted and maybe even on a private or paid relay so you can make it as big as you want
even if you can make it as big as you want, it's still ugly to have to uploads megabytes of data every time you follow or unfollow someone.
Agree with @jb55 . We have many scenarios where we need large lists (we currently don't have that many users, but we might in the future), such as 10k+ follows or 10k+ group members.
It would be great if we could provide incremental updates to the list, such as offering "add follow" and "remove follow" instead of uploading an entire list. For large lists, we can offer segmented downloads or HTTP/HTTPS resume capability for downloads.
Why no one ever considers the possibility of storing follow lists locally, or using Google cloud or something like that? And then publishing those to relays only selectively and/or every once in a while?
I understand the argument of having your contact lists available everywhere around Nostr and that's a good feature, but to think as that as the only good ideal goal to have is probably a symptom of us all being early adopters; still there can be multiple ways of doing things and I know there are some people that would rather not make their follows public.
EDIT: I believe Gossip does this, and Nostur used to do it too.
Now I'm thinking that maybe we could have a protocol for updating lists in relays, like
["APPEND", "<event-id-to-modify>", ["p", "<pubkey>", "whatever"], "<new-id>", "<new-sig>"]`
Has anyone ever suggested that?
Has anyone ever suggested that?
Gotta update created_at as well.
Why "complicate" relays with custom APPEND/REMOVE commands? Relay codebase is quite simple and we still have only rudimentary codebases. I fear that adding many more custom commands to complete the set of all possible changes to events will just complicate things further and block them from working on the things that truly need work in the relay codebase: SYNC, Indexing, and multi-party CDN-like global distribution.
Well, this would be strictly optional. Clients would fall back to just sending the full updated event.
I don't like it either, but being optional I think it might be a better idea than changing the way we do contact lists. But I don't know yet, the idea just came to my mind and might be garbage.
Also relay developers are too lazy. Writing a client is 100x harder than writing a relay. We gotta give them something else to do.
really? it's pretty common on twitter for accounts with 10k+ follows. people have the expectation that they can follow as many people as they want. We can say "too bad", but it feels like we are just giving up and working around a bad design to begin with.
My main reason against large follow lists is that I prefer to preserve how easy it is today for clients to build a Web of Trust with just kind 3s.
People can still follow 10k+ if they want but then I'd rather have it not pollute the current kind 3 space, better keep it local, private or a different kind, or else it will become difficult for clients to deal with and we will end up having to use a few centralized caching services.
On Mon, May 27, 2024 at 02:58:57PM GMT, Fabian wrote:
really? it's pretty common on twitter for accounts with 10k+ follows. people have the expectation that they can follow as many people as they want. We can say "too bad", but it feels like we are just giving up and working around a bad design to begin with.
My main reason against large follow lists is that I prefer to preserve how easy it is today for clients to build a Web of Trust with just kind 3s.
web of trust with kind 3 is already kind of dubious. The replaceable event per pubkey has the benefit of allowing you to refine what the other person is to you:
-
do you want to recieve notes from this person in your timeline?
-
do you prioritize DMs from this person?
-
do you trust this person's 1984 reports so your client can make automatic decisions from their reports on your behalf (I don't trust random people I follow's 1984 reports)
-
general wot rating of this person for use in public algos
I'm not sure if there is a NIP for this idea yet, I can't remember who came up with it. I think it was maybe semisol.