[BUG] Crash

[TimidLo]

• [X] I am reporting a bug.
• [X] I am running the latest version of BDfR
• [X] I have read the Opening an issue

Description

BDFR crashed after a day of running on multiple subreddits to download content with the clone command. Not sure what this crash is about as many items were successfully downloaded over the past day

Command

bdfr clone ./subreddits --opts options.yaml --exclude-id-file success.txt

Environment (please complete the following information)

MacOS Ventura Python 3.10.9

Logs

[2023-01-31 04:17:27,718 - root - ERROR] - Scraper exited unexpectedly
Traceback (most recent call last):
  File "/Users/timidlo/bulk-downloader-for-reddit/venv/bin/bdfr", line 8, in <module>
    sys.exit(cli())
  File "/Users/timidlo/bulk-downloader-for-reddit/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/timidlo/bulk-downloader-for-reddit/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/timidlo/bulk-downloader-for-reddit/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/timidlo/bulk-downloader-for-reddit/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/timidlo/bulk-downloader-for-reddit/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/timidlo/bulk-downloader-for-reddit/venv/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/timidlo/bulk-downloader-for-reddit/venv/lib/python3.10/site-packages/bdfr/__main__.py", line 128, in cli_clone
    reddit_scraper.download()
  File "/Users/timidlo/bulk-downloader-for-reddit/venv/lib/python3.10/site-packages/bdfr/cloner.py", line 21, in download
    for submission in generator:
  File "/Users/timidlo/bulk-downloader-for-reddit/venv/lib/python3.10/site-packages/praw/models/listing/generator.py", line 63, in __next__
    self._next_batch()
  File "/Users/timidlo/bulk-downloader-for-reddit/venv/lib/python3.10/site-packages/praw/models/listing/generator.py", line 89, in _next_batch
    self._listing = self._reddit.get(self.url, params=self.params)
  File "/Users/timidlo/bulk-downloader-for-reddit/venv/lib/python3.10/site-packages/praw/util/deprecate_args.py", line 43, in wrapped
    return func(**dict(zip(_old_args, args)), **kwargs)
  File "/Users/timidlo/bulk-downloader-for-reddit/venv/lib/python3.10/site-packages/praw/reddit.py", line 634, in get
    return self._objectify_request(method="GET", params=params, path=path)
  File "/Users/timidlo/bulk-downloader-for-reddit/venv/lib/python3.10/site-packages/praw/reddit.py", line 739, in _objectify_request
    self.request(
  File "/Users/timidlo/bulk-downloader-for-reddit/venv/lib/python3.10/site-packages/praw/util/deprecate_args.py", line 43, in wrapped
    return func(**dict(zip(_old_args, args)), **kwargs)
  File "/Users/timidlo/bulk-downloader-for-reddit/venv/lib/python3.10/site-packages/praw/reddit.py", line 941, in request
    return self._core.request(
  File "/Users/timidlo/bulk-downloader-for-reddit/venv/lib/python3.10/site-packages/prawcore/sessions.py", line 330, in request
    return self._request_with_retries(
  File "/Users/timidlo/bulk-downloader-for-reddit/venv/lib/python3.10/site-packages/prawcore/sessions.py", line 266, in _request_with_retries
    raise self.STATUS_EXCEPTIONS[response.status_code](response)
prawcore.exceptions.NotFound: received 404 HTTP response

[tiddu]

yes. it's not working since yesterday

[OMEGARAZER]

BDFR crashed after a day of running on multiple subreddits to download content with the clone command.

If it was a single run going that long it's possible the 404 errors came up naturally between the start of the run and when it reached the requested post ending up with the 404.

I had figured this error should have been caught by #713 as well but that appears not to be the case.

yes. it's not working since yesterday

Can you provide more information? All our tests passed on my most recent commit to a PR so if you're having this same issue can you please provide a log file from a failed run so that it can be looked into?

[Serene-Arc]

We really should add a logging message that prints the version of the BDFR being used.