Large channels (>150k messages) fail to jump to search/jump to bottom

[Seb-135]

Expected Behavior

On channels with fewer messages, the client immediately jumps to the latest message archived. When using the search feature and clicking on a message to jump to, the client loads the message and its context immediately. This should occur regardless of number of messages in the channel.

Actual Behavior

For channels around 100k-140k messages, the client visibly stutters and stays blank for a second. The logs show a few (~4) http requests to the messages endpoint. For larger channels, around 150k+, the channel is unusable, as the client starts at the very top of the channel and attempts to continuously scroll down, spamming post requests to the server. Changing the "default scroll position" to "top" prevents this, and allows scrolling normally, but the bottom of the channel is still completely unreachable. Jumping to searched messages results in the same behaviour as clicking on the channel with the "bottom" scroll position - incredibly long waiting times as the client scrolls through messages.

Steps to reproduce the problem

1. Archive a channel with at least 150k messages (a long-lived channel, or any public server's general, should have enough).
2. Start DCEF.
3. Attempt to open the channel, or jump to messages in it via search. Possibly relevant - my export is on a HDD.

Specifications

Environment

• Operating system: Linux Mint 21
• CPU Architecture: x86_64 (PC)
• Flavor: Docker

DiscordChatExporter-frontend

• Release version or commit hash: v2.9.0
• Can the issue be reproduced in the demo?: No (insufficient number of messages)

DiscordChatExporter

• Version (CLI --version): v2.39.1 (I don't have such large backups from v2.40+)

How did you export the data?

• Export format (CLI --format): Json
• Downloaded assets (CLI --media): Yes
• Markdown processing (CLI --markdown): No
• Other CLI options (optional): --reuse-media

Logs

Is there any relevant information in logs? If so, please provide them:

Possibly relevant, for a channel of 140k messages, just clicking on the channel for the first time with scroll position set to "bottom" results in multiple (2-5) post message requests, where smaller channels only request once:

INFO:     127.0.0.1:0 - "GET /search-categories HTTP/1.1" 200 OK
get_message_ids() cache hit - channel id 000000xxxxxxxxxxxxxxxxxx
INFO:     127.0.0.1:0 - "GET /message-ids?guild_id=000000xxxxxxxxxxxxxxxxxx&channel_id=000000xxxxxxxxxxxxxxxxxx HTTP/1.1" 200 OK
INFO:     127.0.0.1:0 - "GET /search-categories HTTP/1.1" 200 OK
INFO:     127.0.0.1:0 - "POST /messages HTTP/1.1" 200 OK
INFO:     127.0.0.1:0 - "POST /messages HTTP/1.1" 200 OK
INFO:     127.0.0.1:0 - "POST /messages HTTP/1.1" 200 OK

Jumping to a searched message in the middle of the channel sends a further 5-6 requests.

The logs for a channel with 180k messages are similar, except the message requests keep being spammed ad nauseam. I got a few hundred before ether the client or server seemingly gave up and stopped requesting/responding to requests, and refused to load more messages.

[slatinsky]

Hello

Thanks for the report. This issue will be fixed in the new rewrite of the frontend, where I will provide alternative "infinite scroll" renderer (like it is in the Discord app). You can see the progress in the newfrontend branch. I won't be fixing this issue in the current version of the frontend.

[slatinsky]

I released the new v3.0.0 version, fixing your issue by using cursor based pagination instead of virtual scroll.