scrapy-playwright
scrapy-playwright copied to clipboard
Published
20 hours ago •
scrapy-plugins
scrapy-plugins
ERROR: Task was destroyed but it is pending!
trafficstars
I sometimes get this error when i use scrapy-pilaywright
2023-03-31 09:33:35 [asyncio] ERROR: Task was destroyed but it is pending!
source_traceback: Object created at (most recent call last):
File "/Users/rccpony/PycharmProjects/bid-spiders/packages/universal_spiders/main.py", line 5, in <module>
cmdline.execute(
File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/scrapy/cmdline.py", line 154, in execute
_run_print_help(parser, _run_command, cmd, args, opts)
File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/scrapy/cmdline.py", line 109, in _run_print_help
func(*a, **kw)
File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/scrapy/cmdline.py", line 162, in _run_command
cmd.run(args, opts)
File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/scrapy/commands/crawl.py", line 27, in run
self.crawler_process.start()
File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/scrapy/crawler.py", line 363, in start
reactor.run(installSignalHandlers=False) # blocking call
File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/twisted/internet/asyncioreactor.py", line 257, in run
self._asyncioEventloop.run_forever()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1901, in _run_once
handle._run()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 249, in run
await self._transport.run()
File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/playwright/_impl/_transport.py", line 155, in run
self.on_message(obj)
File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 210, in <lambda>
self._transport.on_message = lambda msg: self.dispatch(msg)
File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 356, in dispatch
object._channel.emit(
File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/pyee/base.py", line 176, in emit
handled = self._call_handlers(event, args, kwargs)
File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/pyee/base.py", line 154, in _call_handlers
self._emit_run(f, args, kwargs)
File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/pyee/asyncio.py", line 48, in _emit_run
coro: Any = f(*args, **kwargs)
File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/playwright/_impl/_browser_context.py", line 131, in <lambda>
lambda params: self._on_request(
File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/playwright/_impl/_browser_context.py", line 485, in _on_request
page.emit(Page.Events.Request, request)
File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/pyee/base.py", line 176, in emit
handled = self._call_handlers(event, args, kwargs)
File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/pyee/base.py", line 154, in _call_handlers
self._emit_run(f, args, kwargs)
File "/Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/pyee/asyncio.py", line 57, in _emit_run
fut: Any = ensure_future(cast(Any, coro), loop=self._loop)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/tasks.py", line 615, in ensure_future
return _ensure_future(coro_or_future, loop=loop)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/tasks.py", line 636, in _ensure_future
return loop.create_task(coro_or_future)
task: <Task pending name='Task-386' coro=<_make_request_logger.<locals>._log_request() running at /Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/scrapy_playwright/handler.py:606> wait_for=<Future pending cb=[Task.task_wakeup()] created at /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py:429> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/pyee/asyncio.py:65, ProtocolCallback.__init__.<locals>.cb() at /Users/rccpony/Library/Caches/pypoetry/virtualenvs/bid-spiders-KPnmHDqb-py3.10/lib/python3.10/site-packages/playwright/_impl/_connection.py:168] created at /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/tasks.py:636>
[scrapy.utils.log] INFO: Scrapy 2.7.0 started (bot: universal_spiders)
[scrapy.utils.log] INFO: Versions: lxml 4.9.2.0, libxml2 2.9.13, cssselect 1.2.0, parsel 1.7.0, w3lib 2.1.1, Twisted 21.7.0, Python 3.10.10 (v3.10.10:aad5f6a891, Feb 7 2023, 08:47:40) [Clang 13.0.0 (clang-1300.0.29.30)], pyOpenSSL 22.1.0 (OpenSSL 3.0.7 1 Nov 2022), cryptography 38.0.4, Platform macOS-13.2.1-arm64-arm-64bi
(Edit) Reproducible example at #233
Encountering the same issue here.
2023-05-24 09:10:35 [scrapy.core.engine] INFO: Spider closed (finished)
2023-05-24 09:10:35 [scrapy-playwright] INFO: Closing download handler
2023-05-24 09:10:35 [scrapy-playwright] INFO: Closing browser
2023-05-24 09:10:35 [scrapy-playwright] INFO: Closing download handler
2023-05-24 09:10:35 [scrapy-playwright] INFO: Closing browser
2023-05-24 09:10:35 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-5112' coro=<_make_request_logger.<locals>._log_request() running at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/scrapy_playwright/handler.py:606> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/pyee/asyncio.py:65, ProtocolCallback.__init__.<locals>.cb() at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/playwright/_impl/_connection.py:168]>
2023-05-24 09:10:35 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-5113' coro=<_make_request_logger.<locals>._log_request() running at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/scrapy_playwright/handler.py:606> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/pyee/asyncio.py:65, ProtocolCallback.__init__.<locals>.cb() at /home/xxx/.pyenv/versions/3.10.4/envs/crawl/lib/python3.10/site-packages/playwright/_impl/_connection.py:168]>
2023-05-24 09:10:35 [asyncio] ERROR: Task was destroyed but it is pending!
...[many more here]
Logs seem to point here (using scrapy-playwright 0.0.26) but when I read this question I started wondering if this could be related too.
A nice starting point would be understanding if this is a scrapy-playwright or playwright kind of issue. What do you think?
This seems to be related to https://github.com/jfhbrook/pyee/issues/120. Keeping a reference to the tasks prevents this in my tests.
Please share a minimal, reproducible example.
This was an occasional issue, the code that caused the problem was removed by me at the time, and i will try to reproduce the problem recently
Using the example from #233 I'm seeing much less of these messages after upgrading pyee from 9.0.4 to 11.0.0.
@elacuesta Any luck on a solution? I have a long-running scraper with this issue as well. It's not consistent and happens intermittently. Sometimes errors in 10 minutes, sometimes after 4 hours.
I just took your advice and upgraded my Pyee version to 11.0, I will try again.
@elacuesta Any luck on a solution?
I don't have plans to work on this, from what I see it's an upstream issue.
I'm getting a ton of these errors all of a sudden. I was running scrapy + playwright without issue for several months. Was this issue raised with Pyee?
For reference, my environment is using: python 3.11.6 scrapy 2.11.0 scrapy-playwright 0.0.33 twisted 22.10.0 pyee 11.0.1
Example logs:
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4511' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4514' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4512' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4515' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4763' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4766' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4764' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4767' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4772' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4773' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4797' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4798' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4809' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4812' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4810' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4813' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4863' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4864' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4869' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4872' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4875' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4870' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4873' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4878' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4881' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4879' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4882' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4884' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4887' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4885' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4888' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4876' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4931' coro=<Page._on_route() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/playwright/_impl/_page.py:243> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/pyee/asyncio.py:69]>
2023-11-13 08:11:44 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-4932' coro=<ScrapyPlaywrightDownloadHandler._make_request_handler.<locals>._request_handler() running at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/site-packages/scrapy_playwright/handler.py:529> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[gather.<locals>._done_callback() at /opt/homebrew/Caskroom/miniforge/base/envs/project/lib/python3.11/asyncio/tasks.py:764]>
@ThinksFast I was never able to solve the issue here. I ended up migrating to scrapy-splash.
@ThinksFast I was never able to solve the issue here. I ended up migrating to scrapy-splash.
Thanks for the follow up! I found out earlier today I was getting this issue due to an exception in my Scrapy pipeline file. The pipeline function to save the data was throwing an exception due to Unique Constraints set up on the table columns. But this exception wasn't getting logged for some reason, it was silent. Anyway, after fixing that, the errors went away for me.