DisnakeDev
Adding thousands of views to a client is extremely slow
Summary
Registering thousands of persistent views on startup takes O(n^2) time because ViewStore.__verify_integrity gets called each time one is added, and that is O(n).
Reproduction Steps
My bot is a polling bot. On startup I load all the polls from the database, then register a persistent view for each poll, so the bot can continue to respond to interactions on old polls.
Minimal Reproducible Code
class Paul(InteractionBot):
# ...
async def __load_polls(self) -> None:
async for poll in Poll.fetch_polls():
self.add_view(PollView(self, poll))
Expected Results
I expect Client.add_view to run in O(1) time, causing my bot to load in linear time.
Actual Results
My bot has grown to have about 22K polls. Now it takes about 30 minutes to start up, though I notice that at the beginning it loads polls quite fast, but as it progresses it gets slower and slower.
This is because Client.add_view runs in O(n) time, since each time it calls ViewStore.__verify_integrity, which iterates over all previously added views.
Intents
None
System Information
- Python v3.13.0-final
- disnake v2.10.1-final
- disnake importlib.metadata: v2.10.1
- aiohttp v3.11.12
- system info: Linux 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64
Checklist
- [x] I have searched the open issues for duplicates.
- [x] I have shown the entire traceback, if possible.
- [x] I have removed my token from display, if visible.
Additional Context
I think a possible solution would be to have a method Client.bulk_add_views which would propagate to a new method ViewStore.add_views which would only call ViewStore.__verify_integrity once.
Hi, I suggest taking a look at this example (and if you really want to go with that take a look at this project, it's still in a WIP phase but makes everything easier, you can also ask questions in the discord channel of that project), views don't scale well, above all if we're talking about persistent views.
The real issue here is that you need to add a new persistent view every time your bot creates a poll, this way of doing it is not sustainable from the limited resource perspective (too much ram wasted just to store views). The links above bring you to a much more nice and clean way of doing that.
Agree with @Snipy7374 here; at that scale, the sustainable solution would likely be to start using listeners instead of individual views.
That being said, Client.add_view being O(n) indeed isn't great. Long-term, I'm hoping to refactor internals to remove the need for ViewStore.__verify_integrity entirely, so I'll keep this open for the time being.
@Enegg My poll bot Paul has many more features than Discord's native polls.
- Discord has a limit of 9 options, whereas Paul is only limited by the number of buttons I can put under a message (about 25 iirc).
- Discord polls have a very limited number of options for their duration with a max duration of 2 weeks, whereas Paul can have an extremely precise duration and can even remain open indefinitely.
- ~Discord polls cannot be closed manually, but Paul allows the creator to close the poll at any point.~
- With Paul you can restrict voting to certain roles and/or users whereas Discord polls don't have such an option. This allows for mods to vote in public for example.
- With Paul you can specify which users and/or roles may view who voted for each option, allowing for completely or partially anonymous polls, whereas Discord polls are all completely public.
- With Paul you can also specify users and/or roles that are allowed to add options to the poll after its creation, allowing for "open ended" polls where users can suggest new options instead of voting for the existing ones.
