ceiba-dl icon indicating copy to clipboard operation
ceiba-dl copied to clipboard

error when useing ceiba-dl get command

Open Wilson1211 opened this issue 4 years ago • 7 comments

$ ceiba-dl get Traceback (most recent call last): File "/usr/local/bin/ceiba-dl", line 243, in exit(0 if args.func(args, config) else 1) File "/usr/local/bin/ceiba-dl", line 141, in run_get end_download_callback=end_callback) File "/usr/local/lib/python3.5/site-packages/ceiba_dl/init.py", line 388, in run download_progress_callback, end_download_callback) File "/usr/local/lib/python3.5/site-packages/ceiba_dl/init.py", line 232, in download_file if not self.download_file(child_path, retry, dcb, ecb): File "/usr/local/lib/python3.5/site-packages/ceiba_dl/init.py", line 232, in download_file if not self.download_file(child_path, retry, dcb, ecb): File "/usr/local/lib/python3.5/site-packages/ceiba_dl/init.py", line 232, in download_file if not self.download_file(child_path, retry, dcb, ecb): File "/usr/local/lib/python3.5/site-packages/ceiba_dl/init.py", line 214, in download_file node = self.vfs.open(path) File "/usr/local/lib/python3.5/site-packages/ceiba_dl/vfs.py", line 40, in open work.fetch() File "/usr/local/lib/python3.5/site-packages/ceiba_dl/vfs.py", line 979, in fetch course_list_row = self.vfs.root.courses.search_course_list(self._sn) File "/usr/local/lib/python3.5/site-packages/ceiba_dl/vfs.py", line 397, in search_course_list return self._course_list_map[sn] KeyError: '0ff983'

Wilson1211 avatar May 02 '20 15:05 Wilson1211

It seems that CEIBA API returns a course which does not exist on https://ceiba.ntu.edu.tw/student/index.php page. 0ff983 is the CEIBA serial number (csn) of the problematic course.

lantw44 avatar May 03 '20 04:05 lantw44

It seems that courses in 旁聽 have the error described above. Is there any approach to bypass the problem?

Wilson1211 avatar May 03 '20 09:05 Wilson1211

It seems that courses in 旁聽 have the error described above.

Since I have no course marked as 旁聽 in my account, I can't reproduce the issue myself. Does this patch work for you?

diff --git a/ceiba_dl/vfs.py b/ceiba_dl/vfs.py
index a5c6c59..56b6c22 100644
--- a/ceiba_dl/vfs.py
+++ b/ceiba_dl/vfs.py
@@ -394,6 +394,8 @@ class RootCoursesDirectory(Directory):
     def search_course_list(self, sn):
         if not hasattr(self, '_course_list_map'):
             self._create_course_list_map()
+        if sn not in self._course_list_map:
+            return []
         return self._course_list_map[sn]
 
 class RootStudentsDirectory(Directory):

Is there any approach to bypass the problem?

You can add the broken directory to the delete_files list in edit section of your configuration file.

lantw44 avatar May 03 '20 09:05 lantw44

It shows the error message:

Traceback (most recent call last): File "/usr/local/bin/ceiba-dl", line 243, in exit(0 if args.func(args, config) else 1) File "/usr/local/bin/ceiba-dl", line 141, in run_get end_download_callback=end_callback) File "/usr/local/lib/python3.5/site-packages/ceiba_dl/init.py", line 388, in run download_progress_callback, end_download_callback) File "/usr/local/lib/python3.5/site-packages/ceiba_dl/init.py", line 214, in download_file node = self.vfs.open(path) File "/usr/local/lib/python3.5/site-packages/ceiba_dl/vfs.py", line 40, in open work.fetch() File "/usr/local/lib/python3.5/site-packages/ceiba_dl/vfs.py", line 982, in fetch if len(course_list_row[6]) > 0: IndexError: list index out of range

Wilson1211 avatar May 03 '20 11:05 Wilson1211

I am not sure if this patch can work around your problem:

diff --git a/ceiba_dl/vfs.py b/ceiba_dl/vfs.py
index a5c6c59..f042aaf 100644
--- a/ceiba_dl/vfs.py
+++ b/ceiba_dl/vfs.py
@@ -394,6 +394,8 @@ class RootCoursesDirectory(Directory):
     def search_course_list(self, sn):
         if not hasattr(self, '_course_list_map'):
             self._create_course_list_map()
+        if sn not in self._course_list_map:
+            return [[]] * 8
         return self._course_list_map[sn]
 
 class RootStudentsDirectory(Directory):

Since I have no access to your data, it is unlikely that I am able to come up with a working patch for you. The entire project is built on undocumented interfaces of NTU CEIBA, so it is impossible for me to make it reliable. Therefore, when you see an issue, it is recommended to debug and fix it yourself and submit the patch after you resolve the issue. If you can't fix it, please use delete_files to ignore the file or the directory causing the issue.

lantw44 avatar May 23 '20 17:05 lantw44

This works for me since 旁聽 courses are just listed on table 2

diff --git a/ceiba_dl/vfs.py b/ceiba_dl/vfs.py
old mode 100644
new mode 100755
index a5c6c59..d5b8ec7
--- a/ceiba_dl/vfs.py
+++ b/ceiba_dl/vfs.py
@@ -353,6 +353,7 @@ class RootCoursesDirectory(Directory):
         course_list_page = self.vfs.request.web('/student/index.php')
         course_list_rows_all = course_list_page.xpath('//table[1]/tr')
         course_list_rows = course_list_rows_all[1:]
+        course_list_rows += course_list_page.xpath('//table[2]/tr')[1:]
         course_list_header_row = course_list_rows_all[0]

b04505009 avatar Dec 08 '20 13:12 b04505009

This works for me since 旁聽 courses are just listed on table 2

Sorry, it doesn't work for me because I have no 旁聽 courses, and the second table has the title '以下為您修習的課程,並未於CEIBA上開設課程網頁' instead, which causes _create_course_list_map to fail because it can't find the CEIBA serial number.

Traceback (most recent call last):
  File "/home/lantw44/devel/ceiba-dl/ceiba-dl", line 243, in <module>
    exit(0 if args.func(args, config) else 1)
  File "/home/lantw44/devel/ceiba-dl/ceiba-dl", line 158, in run_ls
    lser.run(sys.stdout, path)
  File "/home/lantw44/devel/ceiba-dl/ceiba_dl/__init__.py", line 433, in run
    self.print_file(output, path, self.recursive)
  File "/home/lantw44/devel/ceiba-dl/ceiba_dl/__init__.py", line 397, in print_file
    node = self.vfs.open(path)
  File "/home/lantw44/devel/ceiba-dl/ceiba_dl/vfs.py", line 27, in open
    self._do_edit()
  File "/home/lantw44/devel/ceiba-dl/ceiba_dl/vfs.py", line 63, in _do_edit
    course.fetch()
  File "/home/lantw44/devel/ceiba-dl/ceiba_dl/vfs.py", line 1089, in fetch
    course_list_row = self.vfs.root.courses.search_course_list(self._sn)
  File "/home/lantw44/devel/ceiba-dl/ceiba_dl/vfs.py", line 397, in search_course_list
    self._create_course_list_map()
  File "/home/lantw44/devel/ceiba-dl/ceiba_dl/vfs.py", line 393, in _create_course_list_map
    assert False
AssertionError

lantw44 avatar Jan 31 '22 07:01 lantw44