kemono-dl
kemono-dl copied to clipboard
Ovear
Skip comments fetch for downloaded posts when enable archive
Hello! Checking if the post should be downloaded seems to be very slow, it processes only one post per second. It was much faster in previous versions.
It seems that this is not the case, the delay between posts is still exactly 1 second. I use these bat commands:
@echo on chcp 65001 call C:\ProgramData\Anaconda3\Scripts\activate.bat C:\ProgramData\Anaconda3 C:\ProgramData\Anaconda3\python.exe "C:\kemono-dl\kemono-dl\kemono-dl.py" --cookies "cookie.txt" --post-timeout 2 --icon --banner --archive "archive.txt" --filename-pattern "[{published}] [{id}] {title}{published} {id} {index} {filename}.{ext}" --inline-filename-pattern "[{published}] [{id}] {title}\inline{published} {id} i{index} {filename}.{ext}" --other-filename-pattern "[{published}] [{id}] {title}{published} {id} {filename}.{ext}" --inline --content --comments --json --links https://kemono.party/fanbox/user/9752301 pause
And --post-timeout 2 does not apply to this, because it is a delay between the currently downloading posts, and it does not affect the already downloaded posts.
Please attach verbose logs here too.
There are no delay related behaviores changed in recent version, I assumed this is network related problem or limitation.
2022-11-07 17:27:48,601:DEBUG:Getting creator json from https://kemono.party/api/creators/ 2022-11-07 17:27:49,155:INFO:Downloading posts from kemono.party.party | fanbox | chickeiii | 9752301 2022-11-07 17:27:49,156:DEBUG:Requesting user json from: https://kemono.party/api/fanbox/user/9752301?o=0 2022-11-07 17:27:50,833:INFO:Skipping: [9752301]_icon.jpeg | File already exists 2022-11-07 17:27:50,835:INFO:Skipping post | post already archived 2022-11-07 17:27:51,751:DEBUG:No comments found for this post. 2022-11-07 17:27:51,755:INFO:Skipping post | post already archived 2022-11-07 17:27:52,752:INFO:Skipping post | post already archived 2022-11-07 17:27:53,757:INFO:Skipping post | post already archived 2022-11-07 17:27:54,760:INFO:Skipping post | post already archived 2022-11-07 17:27:55,755:INFO:Skipping post | post already archived 2022-11-07 17:27:56,751:INFO:Skipping post | post already archived 2022-11-07 17:27:57,755:INFO:Skipping post | post already archived 2022-11-07 17:27:58,770:INFO:Skipping post | post already archived 2022-11-07 17:27:59,762:INFO:Skipping post | post already archived 2022-11-07 17:28:00,743:INFO:Skipping post | post already archived
No idea what's causing it. I thought it might be because of the large size of the "archive.txt" file, but even with an almost empty one there is still a 1 second delay.
I found a bug! The --comments function causes a 1 second delay, without it everything works properly.
Is it possible to make the program does not check comments on already downloaded posts? Some creators leave download links in comments, and I use Notepad++ to search for links in posts and comments.
Hi, Thanks for the report.
In this situation, this is more likely a network related problem. The program needs to perform network requests to fetch comments due to the limitation of kemono itself
Since it's the original behaviors which impacts many things, and it seems no quick approach to optimize this. I will consider this as a feature improvement, and it may take some time.
In the previous version, kemono-dl 2022.02.26, the program created 2 files, "content.html" and "comments.html", and they were skipped without problems. https://github.com/AlphaSlayer1964/kemono-dl/releases/tag/2022.02.26
These are two different cases, this branch is based on the latest kemono-dl. There is no 'comments.html' now.
I use the --comments and --archive options, but it still checks every post with a small delay. Is this feature request still possible?
