Same file with identical content gets downloaded again every time Moodle-DL is executed

[TheBlueKingLP]

Description of the bug

There is a file in my moodle account that kept getting downloaded again every time I execute Moodle-DL. The old file then renamed to append _old_01.md For example: XXX_old_28.md XXX_old_27.md XXX_old_26.md

Steps to reproduce the issue

no argument

2024-01-21 21:42:08  DEBUG  {task}  [0] Starting Task: Task (0, File (module_id: 1, section_name: "Topic 1", section_id: "<id are the same>", module_name: "Lesson 1 exer", content_filepath: /, content_filename: "Lesson 1 exer", content_fileurl: "", content_filesize: 1790, content_timemodified: 0, module_modname: label, content_type: description, content_isexternalfile: False, saved_to: "", time_stamp: 0, modified: True, moved: False, deleted: False, notified: False, hash: <all hash are different>, file_id: None, old_file_id: None), Course (id: 5845, fullname: "Course Title", overwrite_name_with: "None", create_directory_structure: True, files: 26), TaskStatus(state=<TaskState.STARTED: 'STARTED'>, bytes_downloaded=0, external_total_size=0, error=None, yt_dlp_failed_with_error=False, yt_dlp_used_generic_extractor=False, yt_dlp_current_file=None, yt_dlp_total_size_per_file={}, yt_dlp_bytes_downloaded_per_file={}))
2024-01-21 21:42:08  DEBUG  {task}  [0] Renaming old file
2024-01-21 21:42:08  DEBUG  {task}  [0] Starting downloading of: Course Title/Topic 1/Lesson 1 exer.md
2024-01-21 21:42:08  DEBUG  {task}  [0] Creating a description file
2024-01-21 21:42:08  DEBUG  {task}  [1] Starting Task: Task (1, File (module_id: 2, section_name: "Topic 1", section_id: "<id are the same>", module_name: "Lesson 1 exer", content_filepath: /, content_filename: "Lesson 1 exer", content_fileurl: "", content_filesize: 1823, content_timemodified: 0, module_modname: label, content_type: description, content_isexternalfile: False, saved_to: "", time_stamp: 0, modified: True, moved: False, deleted: False, notified: False, hash: <all hash are different>, file_id: None, old_file_id: None), Course (id: 5845, fullname: "Course Title", overwrite_name_with: "None", create_directory_structure: True, files: 26), TaskStatus(state=<TaskState.STARTED: 'STARTED'>, bytes_downloaded=0, external_total_size=0, error=None, yt_dlp_failed_with_error=False, yt_dlp_used_generic_extractor=False, yt_dlp_current_file=None, yt_dlp_total_size_per_file={}, yt_dlp_bytes_downloaded_per_file={}))
2024-01-21 21:42:08  DEBUG  {task}  [1] Renaming old file
2024-01-21 21:42:08  DEBUG  {task}  [1] Starting downloading of: Course Title/Topic 1/Lesson 1 exer_01.md
2024-01-21 21:42:08  DEBUG  {task}  [1] Creating a description file
2024-01-21 21:42:08  DEBUG  {task}  [2] Starting Task: Task (2, File (module_id: 3, section_name: "Topic 1", section_id: "<id are the same>", module_name: "Lesson 2 exer", content_filepath: /, content_filename: "Lesson 2 exer", content_fileurl: "", content_filesize: 1802, content_timemodified: 0, module_modname: label, content_type: description, content_isexternalfile: False, saved_to: "", time_stamp: 0, modified: True, moved: False, deleted: False, notified: False, hash: <all hash are different>, file_id: None, old_file_id: None), Course (id: 5845, fullname: "Course Title", overwrite_name_with: "None", create_directory_structure: True, files: 26), TaskStatus(state=<TaskState.STARTED: 'STARTED'>, bytes_downloaded=0, external_total_size=0, error=None, yt_dlp_failed_with_error=False, yt_dlp_used_generic_extractor=False, yt_dlp_current_file=None, yt_dlp_total_size_per_file={}, yt_dlp_bytes_downloaded_per_file={}))
2024-01-21 21:42:08  DEBUG  {task}  [2] Renaming old file
2024-01-21 21:42:08  DEBUG  {task}  [2] Starting downloading of: Course Title/Topic 1/Lesson 2 exer.md
2024-01-21 21:42:08  DEBUG  {task}  [2] Creating a description file
2024-01-21 21:42:08  DEBUG  {task}  [3] Starting Task: Task (3, File (module_id: 5, section_name: "Topic 1", section_id: "<id are the same>", module_name: "Lesson 2 exer", content_filepath: /, content_filename: "Lesson 2 exer", content_fileurl: "", content_filesize: 1779, content_timemodified: 0, module_modname: label, content_type: description, content_isexternalfile: False, saved_to: "", time_stamp: 0, modified: True, moved: False, deleted: False, notified: False, hash: <all hash are different>, file_id: None, old_file_id: None), Course (id: 5845, fullname: "Course Title", overwrite_name_with: "None", create_directory_structure: True, files: 26), TaskStatus(state=<TaskState.STARTED: 'STARTED'>, bytes_downloaded=0, external_total_size=0, error=None, yt_dlp_failed_with_error=False, yt_dlp_used_generic_extractor=False, yt_dlp_current_file=None, yt_dlp_total_size_per_file={}, yt_dlp_bytes_downloaded_per_file={}))
2024-01-21 21:42:08  DEBUG  {task}  [3] Renaming old file
2024-01-21 21:42:08  DEBUG  {task}  [3] Starting downloading of: Course Title/Topic 1/Lesson 2 exer_02.md
2024-01-21 21:42:08  DEBUG  {task}  [3] Creating a description file
2024-01-21 21:42:08  DEBUG  {task}  [4] Starting Task: Task (4, File (module_id: 4, section_name: "Topic 1", section_id: "<id are the same>", module_name: "Lesson 2 exer", content_filepath: /, content_filename: "Lesson 2 exer", content_fileurl: "", content_filesize: 1769, content_timemodified: 0, module_modname: label, content_type: description, content_isexternalfile: False, saved_to: "", time_stamp: 0, modified: True, moved: False, deleted: False, notified: False, hash: <all hash are different>, file_id: None, old_file_id: None), Course (id: 5845, fullname: "Course Title", overwrite_name_with: "None", create_directory_structure: True, files: 26), TaskStatus(state=<TaskState.STARTED: 'STARTED'>, bytes_downloaded=0, external_total_size=0, error=None, yt_dlp_failed_with_error=False, yt_dlp_used_generic_extractor=False, yt_dlp_current_file=None, yt_dlp_total_size_per_file={}, yt_dlp_bytes_downloaded_per_file={}))
2024-01-21 21:42:08  DEBUG  {task}  [4] Renaming old file
2024-01-21 21:42:08  DEBUG  {task}  [4] Starting downloading of: Course Title/Topic 1/Lesson 2 exer_01.md
2024-01-21 21:42:08  DEBUG  {task}  [4] Creating a description file
2024-01-21 21:42:08  DEBUG  {task}  [1] Download finished
2024-01-21 21:42:08  DEBUG  {task}  [0] Download finished
2024-01-21 21:42:08  DEBUG  {task}  [2] Download finished
2024-01-21 21:42:08  DEBUG  {task}  [3] Download finished
2024-01-21 21:42:08  DEBUG  {task}  [4] Download finished

It says there are files that changed but the file is actually identical with no change in content. See anonymized output below

14 changes found for the configured Moodle-Account.
Course Title
<file that are not moved to _old_XX.md are redacted>
≠       Course Title/Topic 1/Lesson 1 exer.md
≠       Course Title/Topic 1/Lesson 3 exer.md
≠       Course Title/Topic 1/Lesson 1 exer_01.md
≠       Course Title/Topic 1/Lesson 2 exer_02.md
≠       Course Title/Topic 1/Lesson 2 exer_01.md
≠       Course Title/Topic 1/Lesson 3 exer_02.md
≠       Course Title/Topic 1/Lesson 3 exer_03.md
≠       Course Title/Topic 1/Lesson 2 exer.md
≠       Course Title/Topic 1/Lesson 3 exer_01.md

Expected behavior

The file is only downloaded once if the content is identical, without moving old file to _old_XX.md

Possible Fix

Technical details

• OS: Arch Linux with kernel 6.7.0-arch3-1
• Moodle-DL Version moodle-dl 2.3.2.0

P.S. if you need a more detailed/anonymized log I can send it to you privately.

[C0D3D3V]

😅 There is probably something changing in the html file (that gets stripped out in markdown). Do you know how to debug python?

[TheBlueKingLP]

Sorry I am not really a programmer. I can code basic scripts but not sophisticated softwares like Moodle-DL. However, I can probably get whatever info you need if instruction is given.

[C0D3D3V]

Please send me a screenshot of your course. Can it be that you have multiple lessons with the same name?

[C0D3D3V]

Mh I just tested it to have two lessons with the same name. For me it works, without redownloading. So I guess it is really something changing in the description of the lessons.

Its pretty funny that moodle adds links to the other lessons with the same name to the description ^^ (at least for one other lesson with the same name). If you have more than two lessons with the same name in the same section the links kind of make no sense. But moodle-dl downloads them correctly. Edit: Correcting myself, moodle does not refer to lessons with same name. That was because I used the name of the lesson in the description. The links were generated by the auot linking feature: https://docs.moodle.org/403/en/Auto-linking

So I need a call with you, so we can debug this together. Maybe on discord. Contact me via mail please

[C0D3D3V]

you could also provide my the files that get always updated (including the old files) maybe I see there what is changing.

I wonder even, what "Lesson 1 exer.md" should be? Because moodle-dl normaly does not create such names.