Feature Request: Either optimizing viewer export, or quick way to export .dht files by server/channel

[bramnet]

trafficstars

I've noticed with DHT, that whenever I save or open a viewer, even if I have the contents filtered to only export one channel, it's dedicating the entire .dht file into memory. This is fine if your .dht file is only a few gigabytes, if even that, but what if your .dht file gets significantly larger, like 10 or 20 GBs. I'm not familiar enough with C# to know if either of these would be easily possible, but it would make things easier for future users if either A: it only dedicated the parts of the .dht file that's needed for export so it doesn't have to potentially spend so much time shoving stuff into the pagefile, or B: give us a way to manually export data from the .dht files as new .dht files so we can keep them a manageable size and we don't have to worry about the program hanging during exports. If there's currently a way to split the .dht file into multiple files, I must be missing something. I see the option to merge, but not to split.

[chylex]

It definitely shouldn't be loading the entire file into memory to export one channel, if you did confirm that's what's happening I don't know if that's an issue with the app or SQLite. SQLite will read through the whole file, but I would assume it's streaming the file. I will investigate eventually.

It should be easy to improve though. There could be an index on the channel so SQLite wouldn't have to read the whole file, and the app could stream messages instead of loading all filtered messages into memory at once.

[chylex]

In the current version, I'm not seeing excessive memory usage when Viewer filters are active.

There have also been various optimizations since this issue was opened, more optimizations are coming in the next update, and eventually #246 will completely eliminate preloading messages into memory, so I'm closing as answered.