botkaca icon indicating copy to clipboard operation
botkaca copied to clipboard

Published 20 hours ago • verified publisher icon azamaulanaaa

BOT CRASHED ON DOWNLOAD DURING LARGE DATA REQUEST

Open talkaal opened this issue 4 years ago • 14 comments

A 38 GB (single files each) file torrent was put but it happened to crash after a while.

As well as two feature request also if possible add!

  • SEEDS/LEECH amount (could be able to estimate health of torrent)
  • Download percentage of Torrents

A minor 🐞Bug

  • Thumbnail generation in video before upload is much slower.

Log of the activity ( HEROKU)

2020-09-29T10:26:57.191869+00:00 app[worker.1]: During handling of the above exception, another exception occurred: 2020-09-29T10:26:57.191869+00:00 app[worker.1]: 2020-09-29T10:26:57.191869+00:00 app[worker.1]: Traceback (most recent call last): 2020-09-29T10:26:57.191870+00:00 app[worker.1]: File "/app/bot/handlers/leech_handler.py", line 110, in progress_dl 2020-09-29T10:26:57.191870+00:00 app[worker.1]: await message.edit( 2020-09-29T10:26:57.191871+00:00 app[worker.1]: File "/usr/local/lib/python3.8/site-packages/pyrogram/client/types/messages_and_media/message.py", line 2507, in edit_text 2020-09-29T10:26:57.191872+00:00 app[worker.1]: return await self._client.edit_message_text( 2020-09-29T10:26:57.191873+00:00 app[worker.1]: File "/usr/local/lib/python3.8/site-packages/pyrogram/client/methods/messages/edit_message_text.py", line 78, in edit_message_text 2020-09-29T10:26:57.191873+00:00 app[worker.1]: r = await self.send( 2020-09-29T10:26:57.191874+00:00 app[worker.1]: File "/usr/local/lib/python3.8/site-packages/pyrogram/client/client.py", line 1424, in send 2020-09-29T10:26:57.191874+00:00 app[worker.1]: r = await self.session.send(data, retries, timeout, self.sleep_threshold) 2020-09-29T10:26:57.191874+00:00 app[worker.1]: File "/usr/local/lib/python3.8/site-packages/pyrogram/session/session.py", line 451, in send 2020-09-29T10:26:57.191875+00:00 app[worker.1]: return await self.send(data, retries - 1, timeout) 2020-09-29T10:26:57.191875+00:00 app[worker.1]: File "/usr/local/lib/python3.8/site-packages/pyrogram/session/session.py", line 429, in send 2020-09-29T10:26:57.191876+00:00 app[worker.1]: return await self._send(data, timeout=timeout) 2020-09-29T10:26:57.191876+00:00 app[worker.1]: File "/usr/local/lib/python3.8/site-packages/pyrogram/session/session.py", line 399, in _send 2020-09-29T10:26:57.191877+00:00 app[worker.1]: RPCError.raise_it(result, type(data)) 2020-09-29T10:26:57.191877+00:00 app[worker.1]: File "/usr/local/lib/python3.8/site-packages/pyrogram/errors/rpc_error.py", line 79, in raise_it 2020-09-29T10:26:57.191878+00:00 app[worker.1]: raise getattr( 2020-09-29T10:26:57.191878+00:00 app[worker.1]: pyrogram.errors.exceptions.bad_request_400.MessageNotModified: [400 MESSAGE_NOT_MODIFIED]: The message was not modified (caused by "messages.EditMessage")

talkaal avatar Sep 29 '20 10:09 talkaal

everything you ask for are possible, it just me don't add it. I will add it soon.

A minor 🐞Bug

  • Thumbnail generation in video before upload is much slower.

for thumbnail is so slow due to ffmpeg needs to seek to the middle of video to generates thumbnail. from many post I read, it is possible to just jump to the middle to snapshot instead of seek but the drawback is sometimes it does not work, in my case mostly does not work. the solution I knew is snapshot the first frame, the drawback it does not represent the video. I still working with it along side optimization for video split.

A 38 GB (single files each) file torrent was put but it happened to crash after a while.

for large file, as far as I know is due to limitation from the heroku itself. I did 5gb+ and most the time the file gone from the service. the app already optimize for binary split ( like hjsplit does) but it just help upto 4-5gb files. I am still working on this issue on my other project torrent-mirror with nodejs since python does not have torrent module natively

azamaulanaaa avatar Sep 29 '20 14:09 azamaulanaaa

CRASHED AGAIN THIS TIME AS PER YOU HAVE TOLD 3GB TORRENT I'VE ADDED.

LOGS

2020-09-29T18:48:17.398735+00:00 app[worker.1]: File "/usr/local/lib/python3.8/email/message.py", line 578, in get_content_type 2020-09-29T18:48:17.398737+00:00 app[worker.1]: value = self.get('content-type', missing) 2020-09-29T18:48:17.398737+00:00 app[worker.1]: File "/usr/local/lib/python3.8/email/message.py", line 471, in get 2020-09-29T18:48:17.398738+00:00 app[worker.1]: return self.policy.header_fetch_parse(k, v) 2020-09-29T18:48:17.398738+00:00 app[worker.1]: File "/usr/local/lib/python3.8/email/_policybase.py", line 316, in header_fetch_parse 2020-09-29T18:48:17.398738+00:00 app[worker.1]: return self._sanitize_header(name, value) 2020-09-29T18:48:17.398739+00:00 app[worker.1]: File "/usr/local/lib/python3.8/email/_policybase.py", line 287, in _sanitize_header 2020-09-29T18:48:17.398739+00:00 app[worker.1]: if _has_surrogates(value): 2020-09-29T18:48:17.398740+00:00 app[worker.1]: File "/usr/local/lib/python3.8/email/utils.py", line 57, in _has_surrogates 2020-09-29T18:48:17.398740+00:00 app[worker.1]: s.encode() 2020-09-29T18:48:17.398741+00:00 app[worker.1]: RecursionError: maximum recursion depth exceeded while calling a Python object 2020-09-29T18:48:17.398741+00:00 app[worker.1]: 2020-09-29T18:48:17.398742+00:00 app[worker.1]: During handling of the above exception, another exception occurred: 2020-09-29T18:48:17.398742+00:00 app[worker.1]: 2020-09-29T18:48:17.398743+00:00 app[worker.1]: Traceback (most recent call last): 2020-09-29T18:48:17.398743+00:00 app[worker.1]: File "/app/bot/handlers/leech_handler.py", line 124, in progress_dl 2020-09-29T18:48:17.398744+00:00 app[worker.1]: return await progress_dl(message, aria2_api, gid, text) 2020-09-29T18:48:17.398745+00:00 app[worker.1]: File "/app/bot/handlers/leech_handler.py", line 139, in progress_dl 2020-09-29T18:48:17.398745+00:00 app[worker.1]: download.remove(force=True) 2020-09-29T18:48:17.398754+00:00 app[worker.1]: UnboundLocalError: local variable 'download' referenced before assignment

Minor UI Alteration if possible

  • Download progress has two "%' symbol

Future feature addition Request (Just a thought)

  • The download progress blocks be extended to 20 rather than currently 10.

  • Add to Specific telegram group for usage rather than individual bot interaction. Multiple users if using the same bot run on the script, the connection to bot will be more and may result in Speed reduction on individual bot interaction.

  • In group this problem might be rectified as group will be a single interactor for request providing files.

  • A command to know available server space and another command to clear residual downloads in the server still remaining this can be used to add torrent after previous file request are uploaded

Thank you for your Interaction based quick response and necessary updates. Hope that I'm not pressurizing you😅

talkaal avatar Sep 30 '20 03:09 talkaal

hi @talkaal, mind if you share the torrent? so I can replicate your bug

azamaulanaaa avatar Sep 30 '20 04:09 azamaulanaaa

  • The download progress blocks be extended to 20 rather than currently 10.

let me give you option to customize it from local/default.py, please wait.

  • Add to Specific telegram group for usage rather than individual bot interaction. Multiple users if using the same bot run on the script, the connection to bot will be more and may result in Speed reduction on individual bot interaction.
  • In group this problem might be rectified as group will be a single interactor for request providing files. you can actually add the bot to specific group.
  1. set the password
  2. add the bot to the group
  3. verify the bot at that group using command /pass [yourpass]
  • A command to know available server space

I can actually do it but there is a catch. Since most service using shared storage, it might not represent actual storage space for you.

another command to clear residual downloads in the server still remaining this can be used to add torrent after previous file request are uploaded

Pardon me, I do not understand.

Note : please create new issue with label enhancement for requested feature. thank you.

azamaulanaaa avatar Sep 30 '20 05:09 azamaulanaaa

hi @talkaal, mind if you share the torrent? so I can replicate your bug

Sure

magnet:?xt=urn:btih:15610ff6ee0d0fe11d491c805cf141235f184e32&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fp4p.arenabg.com%3A1337%2Fannounce&tr=udp%3A%2F%2F9.rarbg.me%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.moeking.me%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.cyberia.is%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker4.itzmx.com%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.internetwarriors.net%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2F9.rarbg.to%3A2710%2Fannounce&tr=udp%3A%2F%2Fexodus.desync.com%3A6969%2Fannounce&tr=udp%3A%2F%2Fexplodie.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fpublic.popcorn-tracker.org%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.vanitycore.co%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker1.itzmx.com%3A8080%2Fannounce&tr=udp%3A%2F%2Fipv4.tracker.harry.lu%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.tiny-vps.com%3A6969%2Fannounce&tr=udp%3A%2F%2Fdenis.stalker.upeer.me%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.torrent.eu.org%3A451%2Fannounce&tr=udp%3A%2F%2Ftracker.port443.xyz%3A6969%2Fannounce&tr=udp%3A%2F%2Fopen.stealth.si%3A80%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.si%3A1337%2Fannounce&tr=udp%3A%2F%2Fbt.xxx-tracker.com%3A2710%2Fannounce&tr=udp%3A%2F%2Fthetracker.org%3A80%2Fannounce&tr=udp%3A%2F%2Fretracker.lanta-net.ru%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.uw0.xyz%3A6969%2Fannounce&dn=%5BSolidTorrents.net%5D+Sense8+Season+2+S02+720p+WEBRip+x265+HEVC+Crazy4ad

talkaal avatar Sep 30 '20 05:09 talkaal

Pardon me, I do not understand.

Since the files are transfered first onto running server and reuploaded to Telegram. There's a chance that the files on server might not delete automatically, so in that case clearing out some space available on server might help.

That's what I meant hope you understood?

talkaal avatar Sep 30 '20 05:09 talkaal

let me give you option to customize it from local/default.py, please wait.

Did you change to customizable variable?

talkaal avatar Sep 30 '20 05:09 talkaal

Since the files are transfered first onto running server and reuploaded to Telegram. There's a chance that the files on server might not delete automatically, so in that case clearing out some space available on server might help.

oh I see, it already been done. the app already optimize for that condition. every files that uploaded will deleted automatically. So if you face some problem after file downloaded. the files might stays there. if you face while downloading, you can cancel them and the files automatically deleted

azamaulanaaa avatar Sep 30 '20 06:09 azamaulanaaa

@talkaal now, you can customize bar size on ./__init__.py. sorry not on local/default.py

azamaulanaaa avatar Sep 30 '20 06:09 azamaulanaaa

hi @talkaal, mind if you share the torrent? so I can replicate your bug

Sure

magnet:?xt=urn:btih:15610ff6ee0d0fe11d491c805cf141235f184e32&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fp4p.arenabg.com%3A1337%2Fannounce&tr=udp%3A%2F%2F9.rarbg.me%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.moeking.me%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.cyberia.is%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker4.itzmx.com%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.internetwarriors.net%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2F9.rarbg.to%3A2710%2Fannounce&tr=udp%3A%2F%2Fexodus.desync.com%3A6969%2Fannounce&tr=udp%3A%2F%2Fexplodie.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fpublic.popcorn-tracker.org%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.vanitycore.co%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker1.itzmx.com%3A8080%2Fannounce&tr=udp%3A%2F%2Fipv4.tracker.harry.lu%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.tiny-vps.com%3A6969%2Fannounce&tr=udp%3A%2F%2Fdenis.stalker.upeer.me%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.torrent.eu.org%3A451%2Fannounce&tr=udp%3A%2F%2Ftracker.port443.xyz%3A6969%2Fannounce&tr=udp%3A%2F%2Fopen.stealth.si%3A80%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.si%3A1337%2Fannounce&tr=udp%3A%2F%2Fbt.xxx-tracker.com%3A2710%2Fannounce&tr=udp%3A%2F%2Fthetracker.org%3A80%2Fannounce&tr=udp%3A%2F%2Fretracker.lanta-net.ru%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.uw0.xyz%3A6969%2Fannounce&dn=%5BSolidTorrents.net%5D+Sense8+Season+2+S02+720p+WEBRip+x265+HEVC+Crazy4ad

looks like the torrent has no seeder

azamaulanaaa avatar Sep 30 '20 13:09 azamaulanaaa

sorry wrong commit reference

azamaulanaaa avatar Oct 04 '20 16:10 azamaulanaaa

The crash of large torrents after a while might be due to two factors of time I think. Don't know whether it's helpful.

Maximum process timeout Maximum time to wait for torrent to start

I hope it'll be a valuable thought. Thank you for your effort on the code

PLEASE DO CLOSE IN CASE THE THOUGHT IS AN INVALID ONE.

talkaal avatar Oct 20 '20 04:10 talkaal

Since the files are transfered first onto running server and reuploaded to Telegram. There's a chance that the files on server might not delete automatically, so in that case clearing out some space available on server might help.

oh I see, it already been done. the app already optimize for that condition. every files that uploaded will deleted automatically. So if you face some problem after file downloaded. the files might stays there. if you face while downloading, you can cancel them and the files automatically deleted

Any chance to youtube download too? 👍

fernandooliveira2 avatar Oct 24 '20 07:10 fernandooliveira2

Since the files are transfered first onto running server and reuploaded to Telegram. There's a chance that the files on server might not delete automatically, so in that case clearing out some space available on server might help.

oh I see, it already been done. the app already optimize for that condition. every files that uploaded will deleted automatically. So if you face some problem after file downloaded. the files might stays there. if you face while downloading, you can cancel them and the files automatically deleted

Any chance to youtube download too? 👍

sorry out of topic

azamaulanaaa avatar Oct 24 '20 12:10 azamaulanaaa