# Files get re-downloaded always

[bebeidon]

Describe the bug

I have the performers folder with all the files and when i wanna download the newest files and choose to "not re-download files" when being asked by the script it will still redownload all files, even when the are already in the folder.

To Reproduce

Steps to reproduce the behavior: I have moved the files outside of the folder before but when i run the script i will put them back in place exactly like they were and this still happens.

Expected behavior

It should only download new files that are not already in the folder.

Screenshots/Logs

https://privatebin.io/?00d65c30613da279#GzachShyL9f8S66PFiZYqwQTqGQrNUTYjBzHcVqvoqFh

Config

{ "main_profile": "main_profile", "metadata": "{configpath}/{profile}/.data/{model_id}", "discord": "", "file_options": { "save_location": "\\NAS\Medien\Video\XXX_tmp\#OnlyFans_ManyVids_etc\#OFScraper", "dir_format": "{model_username}/{responsetype}/{mediatype}/", "file_format": "{filename}.{ext}", "textlength": 0, "space_replacer": " ", "date": "MM-DD-YYYY", "text_type_default": "letter", "truncation_default": true }, "download_options": { "filter": [ "Videos" ], "auto_resume": true, "system_free_min": 0, "max_post_count": 0 }, "binary_options": { "ffmpeg": "" }, "cdm_options": { "private-key": null, "client-id": null, "key-mode-default": "cdrm", "keydb_api": "" }, "performance_options": { "download_sems": 6, "thread_count": 2, "download_limit": 0 }, "content_filter_options": { "block_ads": false, "file_size_max": 0, "file_size_min": 0, "length_max": null, "length_min": null }, "advanced_options": { "code-execution": false, "dynamic-mode-default": "datawhores", "backend": "aio", "downloadbars": false, "cache-mode": "sqlite", "appendlog": true, "custom_values": null, "sanitize_text": false, "temp_dir": null, "remove_hash_match": null, "infinite_loop_action_mode": false, "enable_auto_after": true, "default_user_list": "main", "default_black_list": "" }, "scripts_options": { "post_download_script": null, "post_script": null }, "responsetype": { "timeline": "Posts", "message": "Messages", "archived": "Archived", "paid": "Messages", "stories": "Stories", "highlights": "Stories", "profile": "Profile", "pinned": "Posts", "streams": "Streams" }, "overwrites": { "audios": {}, "videos": {}, "images": {}, "text": {} } }

System Info

• Win10
• Firefox
• Version 3.12.9
• python
• pip

[cjb900]

I'm not experiencing this issue here and haven't seen anyone else mention this recently ether. What command are you using when you run ofscraper?

[bebeidon]

just ofscraper but it seems to work like expected when i add the --after 2000. does the script check against the files that are there or some DB it created?

[cjb900]

Yes, there is a model DB file that is created which logs everything getting scraped from that model so it should only scrape new content.

[WyohKnott]

This commit I authored should fix it:

https://github.com/datawhores/OF-Scraper/commit/d185f62d39bbbbe2fe5d64355c26cb691303da98

Either wait for a new stable release, install the developement branch, or edit the file manually on your system.

[bebeidon]

ah i see thank you. so does it only check against the DB or also against the files? Like i said i moved the files outside of the folder at some point and probably when i was using a older version of the script too. now after idk maybe 1 year i come back and subscribe to a model again and scrape. if i make sure the files are there but there is no DB entries(because i used a old version before probably) what will happen? I was hoping it would be enough if the files are there without any DB updates etc.

[WyohKnott]

ah i see thank you. so does it only check against the DB or also against the files? Like i said i moved the files outside of the folder at some point and probably when i was using a older version of the script too. now after idk maybe 1 year i come back and subscribe to a model again and scrape. if i make sure the files are there but there is no DB entries(because i used a old version before probably) what will happen? I was hoping it would be enough if the files are there without any DB updates etc.

If you move files, they won't get redownloaded because it checks against the databases in the meta directory (depending where you saved them. It is set with the config parameter metadata, for exemple, metadata": "ofscraper2/meta/{model_username}_{model_id}/"). If you moved the metadata folder or the databases, things will get redownloaded because ofscrapper won't find the db and recreate them. You can also force redownloading with --after 2000 --force-all too.

[bebeidon]

that's what i mean what if the DB is not updated because it was a old version of ofscraper i downloaded the files with.? now i have a new version installed without DB entries but the files are there. so will it always download again even if the files are already there if the DB is empty?

[WyohKnott]

that's what i mean what if the DB is not updated because it was a old version of ofscraper i downloaded the files with.? now i have a new version installed without DB entries but the files are there. so will it always download again even if the files are already there if the DB is empty?

yeah, I think so. But there is a metadata command to reconstruct your databases:

ofscraper metadata --help                                                                                                                           38s 439ms  fish 
Usage: OF-Scraper metadata [OPTIONS]

  Uses API to modify db files without the need for downloading

metadata_options:
  -md, --metadata METADATA MODE   
                                  
                                  METADATA MODES
                                  
                                  check:
                                  
                                  Updates metadata fields for the specified area
                                  via an API
                                  
                                  This option does not check for new filenames
                                  
                                  update: This mode performs two actions:
                                  
                                  1. Updates download status and filenames based
                                  on the presence of actual files
                                  
                                  2. Updates metadata fields for the specified
                                  area via an API
                                  
                                  complete:
                                  
                                  Marks the download as complete and updates
                                  metadata fields via an API
                                  
                                  It also uses a new filename if one is
                                  available
  -ms, --mark-stray-locked, --mark-stray
                                  Sets unmatched media items as locked
                                  This is done per api type excluding labels, 
                                  and is limited to --after and --before range
  -an, --anon                     Use anonymise creditional
                                  forces --individual flag
                                  force --list flag to be removed
                                  most media is ignored

Try running ofscraper metadata -md update assuming the files are still correctly located, or ofscraper metadata -md complete followed by options on which models you want the metadata to be be filed if the files have been moved.

[bebeidon]

thank you this is very helpful! :)