[BUG] DuplicateReplaceException: A duplicate comment has been detected. Are you attempting to call

[TrollBlox]

• [ x] I am reporting a bug.
• [x ] I am running the latest version of BDfR
• [ x] I have read the Opening an issue

Description

A clear and concise description of what the bug is.

When downloading, the command hangs for around a minute then spits out this:

  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\Scripts\bdfr.exe\__main__.py", line 7, in <module>
  File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\bdfr\__main__.py", line 161, in cli_clone
    reddit_scraper.download()
  File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\bdfr\cloner.py", line 27, in download
    self.write_entry(submission)
  File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\bdfr\archiver.py", line 92, in write_entry
    self._write_entry_json(archive_entry)
  File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\bdfr\archiver.py", line 103, in _write_entry_json
    content = json.dumps(entry.compile())
  File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\bdfr\archive_entry\comment_archive_entry.py", line 19, in compile
    self.post_details = self._convert_comment_to_dict(self.source)
  File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\bdfr\archive_entry\base_archive_entry.py", line 36, in _convert_comment_to_dict
    in_comment.replies.replace_more(limit=None)
  File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\praw\util\deprecate_args.py", line 43, in wrapped
    return func(**dict(zip(_old_args, args)), **kwargs)
  File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\praw\models\comment_forest.py", line 195, in replace_more
    self._insert_comment(comment)
  File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\praw\models\comment_forest.py", line 80, in _insert_comment
    raise DuplicateReplaceException
praw.exceptions.DuplicateReplaceException: A duplicate comment has been detected. Are you attempting to call 'replace_more_comments' more than once?```

## Command

bdfr clone "C:\Users\caelb\Videos\other stuff\reddit archive\upvoted posts" --user me --saved --authenticate --file-scheme "r/{SUBREDDIT}u/{REDDITOR}{TITLE}_{POSTID}"

## Environment (please complete the following information)

- OS: Windows 11
- Python version: 3.10.11

## Logs

Traceback (most recent call last): File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 86, in run_code exec(code, run_globals) File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\Scripts\bdfr.exe_main.py", line 7, in File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\click\core.py", line 1130, in call return self.main(*args, **kwargs) File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\click\core.py", line 1055, in main rv = self.invoke(ctx) File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\click\core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\click\core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\click\core.py", line 760, in invoke return _callback(*args, **kwargs) File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\click\decorators.py", line 26, in new_func return f(get_current_context(), *args, **kwargs) File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\bdfr_main.py", line 161, in cli_clone reddit_scraper.download() File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\bdfr\cloner.py", line 27, in download self.write_entry(submission) File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\bdfr\archiver.py", line 92, in write_entry self._write_entry_json(archive_entry) File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\bdfr\archiver.py", line 103, in _write_entry_json content = json.dumps(entry.compile()) File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\bdfr\archive_entry\comment_archive_entry.py", line 19, in compile self.post_details = self._convert_comment_to_dict(self.source) File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\bdfr\archive_entry\base_archive_entry.py", line 36, in _convert_comment_to_dict in_comment.replies.replace_more(limit=None) File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\praw\util\deprecate_args.py", line 43, in wrapped return func(**dict(zip(_old_args, args)), **kwargs) File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\praw\models\comment_forest.py", line 195, in replace_more self._insert_comment(comment) File "C:\Users\caelb\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\praw\models\comment_forest.py", line 80, in _insert_comment raise DuplicateReplaceException praw.exceptions.DuplicateReplaceException: A duplicate comment has been detected. Are you attempting to call 'replace_more_comments' more than once?

[TrollBlox]

After running with --log, the only new info I got was [2023-04-21 22:38:45,553 - root - ERROR] - Scraper exited unexpectedly right before it crashes.

[Serene-Arc]

There should be more information before this, like what submission caused this. Please post all of the logs, not just the traceback.

[cheese529]

got this same error today downloading my saved posts with clone command. [2023-05-13 07:01:09,968 - bdfr.downloader - INFO] - Downloaded submission smrp5l from jessicabiel [2023-05-13 07:01:10,174 - bdfr.archiver - INFO] - Record for entry item smrp5l written to disk [2023-05-13 07:01:10,535 - bdfr.downloader - INFO] - Downloaded submission 1sjmf3 from gentlemanboners [2023-05-13 07:01:10,887 - bdfr.archiver - INFO] - Record for entry item 1sjmf3 written to disk [2023-05-13 07:01:10,888 - bdfr.downloader - WARNING] - j6es7x3 is not a submission Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "C:\Users\Mohammad Noor\AppData\Local\Programs\Python\Python311\Scripts\bdfr.exe\__main__.py", line 7, in <module> File "C:\Users\Mohammad Noor\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 1130, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Mohammad Noor\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 1055, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "C:\Users\Mohammad Noor\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Mohammad Noor\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Mohammad Noor\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 760, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Mohammad Noor\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\decorators.py", line 26, in new_func return f(get_current_context(), *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Mohammad Noor\AppData\Local\Programs\Python\Python311\Lib\site-packages\bdfr\__main__.py", line 161, in cli_clone reddit_scraper.download() File "C:\Users\Mohammad Noor\AppData\Local\Programs\Python\Python311\Lib\site-packages\bdfr\cloner.py", line 27, in download self.write_entry(submission) File "C:\Users\Mohammad Noor\AppData\Local\Programs\Python\Python311\Lib\site-packages\bdfr\archiver.py", line 92, in write_entry self._write_entry_json(archive_entry) File "C:\Users\Mohammad Noor\AppData\Local\Programs\Python\Python311\Lib\site-packages\bdfr\archiver.py", line 103, in _write_entry_json content = json.dumps(entry.compile()) ^^^^^^^^^^^^^^^ File "C:\Users\Mohammad Noor\AppData\Local\Programs\Python\Python311\Lib\site-packages\bdfr\archive_entry\comment_archive_entry.py", line 19, in compile self.post_details = self._convert_comment_to_dict(self.source) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Mohammad Noor\AppData\Local\Programs\Python\Python311\Lib\site-packages\bdfr\archive_entry\base_archive_entry.py", line 36, in _convert_comment_to_dict in_comment.replies.replace_more(limit=None) File "C:\Users\Mohammad Noor\AppData\Local\Programs\Python\Python311\Lib\site-packages\praw\util\deprecate_args.py", line 43, in wrapped return func(**dict(zip(_old_args, args)), **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Mohammad Noor\AppData\Local\Programs\Python\Python311\Lib\site-packages\praw\models\comment_forest.py", line 195, in replace_more self._insert_comment(comment) File "C:\Users\Mohammad Noor\AppData\Local\Programs\Python\Python311\Lib\site-packages\praw\models\comment_forest.py", line 80, in _insert_comment raise DuplicateReplaceException praw.exceptions.DuplicateReplaceException: A duplicate comment has been detected. Are you attempting to call 'replace_more_comments' more than once?

[cheese529]

seems like the issue arises when you have a saved comment, bdfr does not see it as a submission and instead of skipping it just terminates the script. @Serene-Arc

[nsc1]

i've triggered the same bug trying to download my saved items, which include a lot of individual comments rather than whole posts. it's not great, but as a workaround i added --comment-context to my command line. it downloads the entire post (which in some cases is massive) instead of just the comment i wanted, but so far it hasn't crashed.

[Serene-Arc]

Alright this should be a pretty easy fix, I'll get onto it when I have some time.

[sulsan]

Alright this should be a pretty easy fix, I'll get onto it when I have some time.

It has something to do with this. All people that had this error had it with a comment starting with 'j' in the comment id. https://www.reddit.com/r/redditdev/comments/119qx01/issues_with_fetching_comment_chains_around_21st/

Edit: nevermind, some other comments have similar issues

[poperigby]

Is there anything I can do to help troubleshoot this? I'm running into the same issue.

[nsc1]

as a workaround you can add --comment-context to your command line. it downloads more than you need, but it doesn't crash.