linkedin-course-downloader
linkedin-course-downloader copied to clipboard
Ahmed-Soli
Downloads getting stuck
Hi, I am finding a new issue, when downloading multiple courses, if the download gets stuck for whatever reasons like connection drop for one second, it gets stuck and does not reattempt.
Is there a way you can make the downloader reattempt if the connection dropped or gets stuck for a prolonged period of time for any reason. Please look into it.
First of all , thank you for giving me the chance of make the script better . Yes there is more than one way I can work around this issue , I would like to know which stage did it stuck on ( download , fetching , writing , etc )
Mine got stuck after a while of downloading successfully and my network dropped due to poor service. I watched it for a while say, 20 minutes and nothing happened, so I closed the terminal. Afterwards, I opened terminal afresh and restarted the download and bam, it worked. But my network did not drop again this time. Thanks for the tool
@iykedairo Glad you like it , yes indeed it picks up from where it left off .
In my case, it just stopped while writing I believe. On my second or third attempt I tried using Windows CMD shell, which can show download progress unlike the gitbash, it appears that it just gets stuck randomly due to connection drop. The only way is it close the terminal, delete the last file that it was attempting which will show 0 bytes in file size, and restart the downloader.py using either gitbash or cmd. If you can write something like, if it gets stuck without movement for like 30 minutes or more, then restart from the previously failed file. It is important that it only starts again from the file it got stuck on, because it would be a waste of bandwidth to redownload the whole courses again because when I put on like 30 courses in the .json file, if the downloader.py starts from the beginning, it would download the entire courses from the beginning. Most of the time I load about 30 courses, and go to bed, when I come back to check, it probably gets stuck after 10 courses, and on a specific video, so we wanna make sure it ignores the successfuly completed downloads in the download folder. I am sorry I am rambling, but I hope I made some sense?? Please let me know if you want me to try again in explaining this. If you can add something like giving us an option to manually say yes to retry. For example if it got stuck in writing after network dropped, file failed, reattempt again? Yes or No. And let the user type in python yes.py or something lol
Comparing this to utorrent, or some torrent client, you know how they always start again from where it left off, even if you hard pressed the power button, when the machine restarts and if there is internet connection, it would restart right from where it left off without losing the data or without downloading again from the beginning. Thanks again so much for your effort Soli. Please also work on downloading the courses based on author's name. lol
Mine got stuck after a while of downloading successfully and my network dropped due to poor service. I watched it for a while say, 20 minutes and nothing happened, so I closed the terminal. Afterwards, I opened terminal afresh and restarted the download and bam, it worked. But my network did not drop again this time. Thanks for the tool
I think closing and restarting the terminal is not a fix for this unfortunately. Because, when you have so many courses being downloaded you don't want to redownload the previously already downloaded files. Maybe it can warn us saying the connection has failed, attempt again? Yes or No option. That way it can continue from where it left off instead of us going in and deleting the half files. I don't want to continue to subscribe to linkedin premium (so expensive) I wanna see if I can get maximum valuable courses that will help my future, get in download and get out lol I have been closing the terminal every few hours to restart the downloads. I can't complain because this is a wonderful tool and it is my poor network in this remote village like area :(
BY THE WAY, you have to delete the last file where it failed before redownloading it because when you close the terminal after getting stuck on lets say a video or an exercise file.zip, and if the file size is originally 1 GB. While downloading it will show the file size to be 0 bytes. When you close the terminal while its stuck, as you close it the file will have some weight to it, like maybe 200 MB, but the actual size is 1 GB right. So when you redownload it, it will skip over this file because its already in the folder eventhough its not properly downloaded yet. So you might want to make sure that all your files are working properly after you closed the terminal.
@iykedairo Glad you like it , yes indeed it picks up from where it left off .
I don't think it does, I mean it skips over fully done files but it doesn't on partially downloaded files. When I close the terminal, the file gets dumped with data which is half done. When I restart the downloader, it particularly skips this half done file as well, and continues on the next video if its not at all downloaded from the beginning. So I manually delete just the half done file, so it won't skip over that file.
@wonderness I get the idea now and I can make some adjustments but unfortunately I'm busy working on other projects right now , but totally noted and will work on it once I get the chance
I am getting these errors. I am not sure where the code is going wrong? Is it because there is no delay in fetching the videos and LinkedIn is blocking the fast downloads?
C:\linkedin-course-downloader-master>downloader.py 2020-02-01 18:49:50,903 root INFO [] -------------Login------------- 2020-02-01 18:49:50,903 root INFO [] Login step 1 - Getting CSRF token... 2020-02-01 18:49:51,218 root INFO [] Login step 1 - Done 2020-02-01 18:49:51,218 root INFO [] Login step 2 - Logging In... 2020-02-01 18:49:53,198 root INFO [] Login step 2 - Done 2020-02-01 18:49:53,204 root INFO [] -------------Done------------- 2020-02-01 18:49:53,204 root INFO [] -------------Fetching Course------------- 2020-02-01 18:49:53,936 root INFO [] Fetching course Learning Ansible Exercise Files 2020-02-01 18:49:53,937 root INFO [*] Fetching course Learning Ansible 2020-02-01 18:49:53,950 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 1 Video no. 1 2020-02-01 18:49:54,381 root INFO [~] Writing 01 - Welcome.mp4 [################################] 20141/20141 - 00:00:03 2020-02-01 18:49:58,223 root INFO [~] Writing 01 - Welcome.srt 2020-02-01 18:49:58,224 root INFO [~] Done fetching course 'Learning Ansible' Chapter no. 1 Video no. 1 2020-02-01 18:49:58,225 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 1 Video no. 2 2020-02-01 18:49:58,860 root INFO [~] Writing 02 - What you should know.mp4 [################################] 2781/2781 - 00:00:00 2020-02-01 18:49:59,386 root INFO [~] Writing 02 - What you should know.srt 2020-02-01 18:49:59,387 root INFO [~] Done fetching course 'Learning Ansible' Chapter no. 1 Video no. 2 2020-02-01 18:49:59,387 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 2 Video no. 1 2020-02-01 18:49:59,799 root INFO [~] Writing 01 - An introduction to Ansible.mp4 [################################] 9977/9977 - 00:00:01 2020-02-01 18:50:01,227 root INFO [~] Writing 01 - An introduction to Ansible.srt 2020-02-01 18:50:01,228 root INFO [~] Done fetching course 'Learning Ansible' Chapter no. 2 Video no. 1 2020-02-01 18:50:01,228 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 2 Video no. 2 2020-02-01 18:50:01,646 root ERROR Extracting Video URL Error , make sure you have access to linkedin learning via premium account 2020-02-01 18:50:01,648 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 2 Video no. 3 2020-02-01 18:50:02,056 root INFO [~] Writing 03 - Get started with Ansible.mp4 [################################] 20680/20680 - 00:00:02 2020-02-01 18:50:05,042 root INFO [~] Writing 03 - Get started with Ansible.srt 2020-02-01 18:50:05,044 root INFO [~] Done fetching course 'Learning Ansible' Chapter no. 2 Video no. 3 2020-02-01 18:50:05,044 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 3 Video no. 1 2020-02-01 18:50:05,619 root ERROR Extracting Video URL Error , make sure you have access to linkedin learning via premium account 2020-02-01 18:50:05,621 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 3 Video no. 2 2020-02-01 18:50:06,108 root ERROR Extracting Video URL Error , make sure you have access to linkedin learning via premium account 2020-02-01 18:50:06,110 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 3 Video no. 3 2020-02-01 18:50:06,641 root ERROR Extracting Video URL Error , make sure you have access to linkedin learning via premium account 2020-02-01 18:50:06,643 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 3 Video no. 4 2020-02-01 18:50:07,133 root ERROR Extracting Video URL Error , make sure you have access to linkedin learning via premium account 2020-02-01 18:50:07,135 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 3 Video no. 5 2020-02-01 18:50:07,546 root ERROR Extracting Video URL Error , make sure you have access to linkedin learning via premium account 2020-02-01 18:50:07,548 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 3 Video no. 6 2020-02-01 18:50:07,946 root ERROR Extracting Video URL Error , make sure you have access to linkedin learning via premium account 2020-02-01 18:50:07,948 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 4 Video no. 1 2020-02-01 18:50:08,476 root ERROR Extracting Video URL Error , make sure you have access to linkedin learning via premium account 2020-02-01 18:50:08,477 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 4 Video no. 2 2020-02-01 18:50:09,434 root ERROR Extracting Video URL Error , make sure you have access to linkedin learning via premium account 2020-02-01 18:50:09,436 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 4 Video no. 3 2020-02-01 18:50:11,136 root ERROR Extracting Video URL Error , make sure you have access to linkedin learning via premium account 2020-02-01 18:50:11,138 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 4 Video no. 4 2020-02-01 18:50:11,572 root ERROR Extracting Video URL Error , make sure you have access to linkedin learning via premium account 2020-02-01 18:50:11,574 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 4 Video no. 5 2020-02-01 18:50:11,983 root ERROR Extracting Video URL Error , make sure you have access to linkedin learning via premium account 2020-02-01 18:50:11,985 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 4 Video no. 6 2020-02-01 18:50:12,505 root ERROR Extracting Video URL Error , make sure you have access to linkedin learning via premium account 2020-02-01 18:50:12,507 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 4 Video no. 7 2020-02-01 18:50:12,932 root ERROR Extracting Video URL Error , make sure you have access to linkedin learning via premium account 2020-02-01 18:50:12,934 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 4 Video no. 8 2020-02-01 18:50:13,342 root ERROR Extracting Video URL Error , make sure you have access to linkedin learning via premium account 2020-02-01 18:50:13,344 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 5 Video no. 1 2020-02-01 18:50:13,824 root ERROR Extracting Video URL Error , make sure you have access to linkedin learning via premium account 2020-02-01 18:50:13,826 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 5 Video no. 2 2020-02-01 18:50:14,304 root ERROR Extracting Video URL Error , make sure you have access to linkedin learning via premium account 2020-02-01 18:50:14,305 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 5 Video no. 3 2020-02-01 18:50:14,803 root ERROR Extracting Video URL Error , make sure you have access to linkedin learning via premium account 2020-02-01 18:50:14,805 root INFO [~] Fetching course 'Learning Ansible' Chapter no. 6 Video no. 1 2020-02-01 18:50:15,281 root INFO [~] Writing 01 - Next steps.mp4
@wonderness I had the exact same issue when my account was premium then the premium ended
My premium account is STILL premium. I can watch the videos, I just can't download them using the script... Does it need a kind of delay? Maybe LinkedIn is recognising that the downloads are too fast and cutting it off?
@wonderness that might be a reason as well , I'm thinking of making a python GUI or a c# app to make it easier and to add more features .
@Ahmed-Soli I am looking forward to that. Hopefully soon, because my premium will end in 1 week, that's why I need to download some videos :(
@Ahmed-Soli are you able to say which part of the code I should add time.sleep(5) to delay the process for 5 seconds between downloads. I am guessing it will be the fetch_info.py file that needs editing. Is that so?
@MojoMoe Yes fetch_info.py add it after line 80 . make sure it has the same indentation as the fetch_course(course)
@Ahmed-Soli Thanks for the response. I have tried your suggestion above and I am still getting the error above. It is not applying the 5 seconds delay :(
@Ahmed-Soli Yes. I get this error 'ERROR Connection Error: name 'time' is not defined'
yep it's because the correct format is from time import sleep try it and let me know
@Ahmed-Soli yes that is what I tried 'from time import sleep' and also 'import time' together and I get this error: ERROR Connection Error: 'elements' If I only add 'from time import sleep', I get this error: ERROR Connection Error: name 'time' is not defined
The errors always start after downloading the 4th video in the course in any of the scenarios, with or without the 5 second delay code.