Describe the bug
After scraping a profile, the error "Broken pipe" appears and subsequent profiles do not get scraped as the error gets thrown after every action. Files seem to be downloaded correctly, but something after the scraping fails and the script will not continue correctly.
This started 1-2 weeks ago. Weirdly, this is not reflected in the logs.
To Reproduce
Scrape any profile or run the scraper with "--username ALL" with multiple subscriptions.
Expected behavior
Normal processing of profiles.
Screenshots/Logs
Traceback (most recent call last):
File "/usr/local/lib/python3.11/logging/handlers.py", line 1496, in emit
self.enqueue(self.prepare(record))
File "/usr/local/lib/python3.11/logging/handlers.py", line 1454, in enqueue
self.queue.put_nowait(record)
File "", line 2, in put_nowait
File "/venv/lib/python3.11/site-packages/multiprocess/managers.py", line 821, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/venv/lib/python3.11/site-packages/multiprocess/connection.py", line 209, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/venv/lib/python3.11/site-packages/multiprocess/connection.py", line 430, in _send_bytes
self._send(header + buf)
File "/venv/lib/python3.11/site-packages/multiprocess/connection.py", line 387, in _send
n = write(self._handle, buf)
^^^^^^^^^^^^^^^^^^^^^^^^
BrokenPipeError: [Errno 32] Broken pipe
Call stack:
File "/venv/bin/ofscraper", line 8, in
sys.exit(main())
File "/venv/lib/python3.11/site-packages/ofscraper/main.py", line 10, in main
load.main()
File "/venv/lib/python3.11/site-packages/ofscraper/runner/load.py", line 23, in main
run.main()
File "/venv/lib/python3.11/site-packages/ofscraper/runner/run.py", line 17, in main
main_helper()
File "/venv/lib/python3.11/site-packages/ofscraper/runner/run.py", line 47, in main_helper
picker.pick()
File "/venv/lib/python3.11/site-packages/ofscraper/commands/picker.py", line 20, in pick
scraper.main()
File "/venv/lib/python3.11/site-packages/ofscraper/commands/scraper.py", line 88, in main
scrapper()
File "/venv/lib/python3.11/site-packages/ofscraper/commands/scraper.py", line 123, in scrapper
process_selected_areas()
File "/venv/lib/python3.11/site-packages/ofscraper/commands/scraper.py", line 41, in process_selected_areas
run.run_helper(*functs)
File "/venv/lib/python3.11/site-packages/ofscraper/utils/run.py", line 109, in run_helper
job_func()
File "/venv/lib/python3.11/site-packages/ofscraper/utils/context/exit.py", line 85, in inner
func(*args, **kwargs)
File "/venv/lib/python3.11/site-packages/ofscraper/actions/process.py", line 71, in process_post
normal_post_process()
File "/venv/lib/python3.11/site-packages/ofscraper/utils/context/exit.py", line 85, in inner
func(*args, **kwargs)
File "/venv/lib/python3.11/site-packages/ofscraper/actions/process.py", line 164, in normal_post_process
combined_urls = OF.process_areas(ele, model_id)
File "/venv/lib/python3.11/site-packages/ofscraper/actions/scraper.py", line 476, in process_areas
profile_dicts = process_profile(username)
File "/venv/lib/python3.11/site-packages/ofscraper/utils/system/free.py", line 16, in inner
return func(*args, **kwargs)
File "/venv/lib/python3.11/site-packages/ofscraper/actions/scraper.py", line 323, in process_profile
user_profile = profile.scrape_profile(username)
File "/venv/lib/python3.11/site-packages/ofscraper/api/profile.py", line 43, in scrape_profile
return scrape_profile_helper(c, username)
File "/venv/lib/python3.11/site-packages/ofscraper/api/profile.py", line 64, in scrape_profile_helper
log.info(
Hint: Please make sure that the log formatting is readable, use a paste site
like https://paste.passtheheadphones.me/
Desktop (please complete the following information):
- OS: Docker / Unraid
- Version 3.7.1
- python
I'm running the latest stable version ofscraper 3.8.16 and facing the same issue, unfortunately:
you can find the terminal response here:
https://paste.passtheheadphones.me/?8f9cbff8bf891fce#2N3VnvKUuJ5q8kz1oFNvvBfUoaNXK7rYbhC9TCkWiq6q
does this still happen on 3.10dev
I upgraded to 3.10.2 and it still happens:
Traceback (most recent call last):
File "/venv/lib/python3.11/site-packages/ofscraper/utils/logs/stdout.py", line 37, in logger_process
messages = funct(timeout=constants.getattr("LOGGER_TIMEOUT"))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 2, in get
File "/venv/lib/python3.11/site-packages/multiprocess/managers.py", line 821, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/venv/lib/python3.11/site-packages/multiprocess/connection.py", line 209, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/venv/lib/python3.11/site-packages/multiprocess/connection.py", line 430, in _send_bytes
self._send(header + buf)
File "/venv/lib/python3.11/site-packages/multiprocess/connection.py", line 387, in _send
n = write(self._handle, buf)
^^^^^^^^^^^^^^^^^^^^^^^^
BrokenPipeError: [Errno 32] Broken pipe
Any idea why? Environment is still the same as mentioned above.
Are you using the official docker images?
Yes, just upgraded to 3.11 with this image:
datawhores/of-scraper:3.11
and the error persists, although now only seems to happen after downloading content from several models in succession.
Doesn't seem to happen when downloading from a single model. Maybe that helps with finding the error.
Problem seems to be solved with version 3.12.9, thanks for the awesome work ;)
datawhores
