Add pagination

[ibizaman]

Description I ran a task and have a lot of files that matched: Photos 153957 files (350.47 GB) (precision: 100%) When opening the task, I see a bunch of web requests to https://my.nextcloud.com/index.php/apps/mediadc/api/v1/tasks/3, and they all time out (> 30 seconds) or do return but the response is huge: content-length: 338923 gzipped. Everything is pretty slow on this page.

I propose to solve this issue by implementing pagination. So the web requests could only load a few items. This would drastically shorten the time the web requests take.

Thank you for this great app!

[andrey18106]

@ibizaman Hi! Thanks for the issue. Yea, it seems like you have a lot of duplicate groups so it becomes too big a web request (which querying the list every 5 seconds) to fit in timeout. I haven't tested it with such an amount, I think I will add batch loading for duplicate groups list in future updates.

For now, you can split your big folder into small ones as a workaround and try to work with duplicates in that way. If the previous task was finished - all image and video hashes would be already computed, so the further scans will work fast.

[ibizaman]

Thanks! I thought about doing the workaround as you said. I… was lazy to try it but it seems like it’s a start. Then when all subfolders are done ill try the root folder again.

[bigcat88]

The problem was partially resolved in version 0-2-0. is it still topical?

[andrey18106]

Closing as stale and partially resolved.

Feel free to reopen and write here if needed.