watchfiles
watchfiles copied to clipboard
samuelcolvin
RuntimeError: Already borrowed since last release when
Description
When exiting my application I'm getting RuntimeError: Already borrowed error
Example Code
No response
Watchfiles Output
future: <Task finished name='Task-9' coro=<Agent.process_door_events() done, defined at /home/bellini/dev/2u/nomad/nomad/agents/base.py:176> exception=RuntimeError('Already borrowed')>
Traceback (most recent call last):
File "/home/bellini/dev/2u/nomad/nomad/agents/base.py", line 181, in process_door_events
async for door_opened in self.listen_for_door_events():
File "/home/bellini/dev/2u/nomad/nomad/agents/dev.py", line 66, in listen_for_door_events
async for changes in watchfiles.awatch(self._incoming_file):
File "/home/bellini/dev/2u/nomad/.venv/lib/python3.10/site-packages/watchfiles/main.py", line 222, in awatch
with RustNotify([str(p) for p in paths], debug, force_polling, poll_delay_ms) as watcher:
RuntimeError: Already borrowed
Operating System & Architecture
Debian Sid amd64
Environment
python3.10
Python & Watchfiles Version
python: 3.10.5 (main, Jun 8 2022, 09:26:22) [GCC 11.3.0], watchfiles: 0.16.0
Rust & Cargo Version
cargo 1.56.0 rustc 1.59.0
Thanks for reporting.
To help narrow down the error, please can you check if you get the same error with v0.15?
Also:
- are you getting the error all the time, or intermittently?
- Is there anything unusual about your setup or anything specific which is causing the error?
To help narrow down the error, please can you check if you get the same error with v0.15?
I don't, the error started happening once I upgraded to v0.16 (sorry, I think I didn't make this clear enough =P)
are you getting the error all the time, or intermittently?
I only get the error when exiting my app. I stop it with SIGINT and I handle it myself, where I cancel the task that is running async for changed in watchfiles.awatch(...)
Is there anything unusual about your setup or anything specific which is causing the error?
Maybe the fact that I handle SIGNINT/KeyboardInterrupt to gracefully exit the application.
Ok, I don't get this and obviously the tests don't pick it up.
Can you share some example code that's causing the error?
@samuelcolvin this example should reproduce the issue is you run it and either ctrl+c or kill -2 <PID> it and the error should happen.
I added some prints to show that the code follows the expected path and it finishes gracefully, but I can see the RuntimeError: Already borrowed at the end.
import asyncio
import signal
import watchfiles
async def watch_files():
async for changes in watchfiles.awatch("/tmp/foobar"):
pass
async def main():
exit_event = asyncio.Event()
loop = asyncio.get_running_loop()
for s in [signal.SIGINT, signal.SIGTERM]:
loop.add_signal_handler(s, exit_event.set)
print("1) Initiating watch_files")
task = asyncio.create_task(watch_files())
await exit_event.wait()
print("2) SIGINT detected, canceling the task")
task.cancel()
done_event = asyncio.Event()
task.add_done_callback(lambda t: done_event.set())
print("3) Waiting for the task to finish")
await done_event.wait()
print("4) task finished")
if __name__ == "__main__":
print("** Before main")
asyncio.run(main())
print("** After main")
Running this on my M1 mac it runs fine without the error. I'll try tomorrow on my linux desktop.
I suspect it's a problem with pyo3 somehow, but I don't want to ask there until we've narrowed it down a bit (or failed to do so).
Running this on my M1 mac it runs fine without the error. I'll try tomorrow on my linux desktop.
I suspect it's a problem with pyo3 somehow, but I don't want to ask there until we've narrowed it down a bit (or failed to do so).
Very strange, what version of python you are running there? I'm running python 3.10.5
Python 3.10.5 😄 .
Just double checking:
- you installed a wheel from pypi?
- you're using libc not musl or something weird?
Yes, I created a fresh venv to run that code and installed watchfiles with pip install watchfiles, which installed the latest wheel
And no weird things, it is the python shipped with debian itself: https://packages.debian.org/sid/python3.10
Works fine for me on Linux. One last thing to check before I report to pyo3. You mentioned you're on amd64, I'm on x86_64, could you just run python -c 'import platform; print(platform.platform()); print(platform.version())' in case that's the difference.
My output:
➤ python -c 'import platform; print(platform.platform()); print(platform.version())'
Linux-5.15.0-41-generic-x86_64-with-glibc2.35
#44-Ubuntu SMP Wed Jun 22 14:20:53 UTC 2022
It's probably just a difference on how Ubuntu and Debian calls, but they should be the same arch.
❯ python3 -c 'import platform; print(platform.platform()); print(platform.version())'
Linux-5.18.0-2-amd64-x86_64-with-glibc2.33
#1 SMP PREEMPT_DYNAMIC Debian 5.18.5-1 (2022-06-16)
And so strange that the issue don't happen to you, it happens literally everytime for me (it is not something that happen "sometimes"). Here is the full testing for reference:
/tmp at 11:06:22
❯ python3 -m venv venv
/tmp took 3s at 11:06:52
❯ source venv/bin/activate
/tmp tmp at 11:06:54
❯ pip install watchfiles
Collecting watchfiles
Using cached watchfiles-0.16.1-cp37-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.2 MB)
Collecting anyio<4,>=3.0.0
Using cached anyio-3.6.1-py3-none-any.whl (80 kB)
Collecting sniffio>=1.1
Using cached sniffio-1.2.0-py3-none-any.whl (10 kB)
Collecting idna>=2.8
Using cached idna-3.3-py3-none-any.whl (61 kB)
Installing collected packages: sniffio, idna, anyio, watchfiles
Successfully installed anyio-3.6.1 idna-3.3 sniffio-1.2.0 watchfiles-0.16.1
/tmp tmp at 11:06:58
❯ python3 test_watchfiles.py
** Before main
1) Initiating watch_files
^C2) SIGINT detected, canceling the task
3) Waiting for the task to finish
4) task finished
** After main
Task exception was never retrieved
future: <Task finished name='Task-2' coro=<watch_files() done, defined at /tmp/test_watchfiles.py:7> exception=RuntimeError('Already borrowed')>
Traceback (most recent call last):
File "/tmp/test_watchfiles.py", line 8, in watch_files
async for changes in watchfiles.awatch("/tmp/foobar"):
File "/tmp/venv/lib/python3.10/site-packages/watchfiles/main.py", line 222, in awatch
with RustNotify([str(p) for p in paths], debug, force_polling, poll_delay_ms) as watcher:
RuntimeError: Already borrowed
I think you should create an issue on pyo3 and see if those guys can help. I think it could be related to
https://github.com/samuelcolvin/watchfiles/blob/7cc903004bbea712a295c6674e728ce27752b0dc/src/lib.rs#L276
But I really don't know.
Thanks for reporting.
To help narrow down the error, please can you check if you get it with v0.15?
On Mon, 25 Jul 2022, 17:59 Thiago Bellini Ribeiro, @.***> wrote:
Description
When exiting my application I'm getting RuntimeError: Already borrowed error Example Code
No response Watchfiles Output
future: <Task finished name='Task-9' coro=<Agent.process_door_events() done, defined at /home/bellini/dev/2u/nomad/nomad/agents/base.py:176> exception=RuntimeError('Already borrowed')> Traceback (most recent call last): File "/home/bellini/dev/2u/nomad/nomad/agents/base.py", line 181, in process_door_events async for door_opened in self.listen_for_door_events(): File "/home/bellini/dev/2u/nomad/nomad/agents/dev.py", line 66, in listen_for_door_events async for changes in watchfiles.awatch(self._incoming_file): File "/home/bellini/dev/2u/nomad/.venv/lib/python3.10/site-packages/watchfiles/main.py", line 222, in awatch with RustNotify([str(p) for p in paths], debug, force_polling, poll_delay_ms) as watcher: RuntimeError: Already borrowed
Operating System & Architecture
Debian Sid amd64 Environment
python3.10 Python & Watchfiles Version
python: 3.10.5 (main, Jun 8 2022, 09:26:22) [GCC 11.3.0], watchfiles: 0.16.0 Rust & Cargo Version
cargo 1.56.0 rustc 1.59.0
— Reply to this email directly, view it on GitHub https://github.com/samuelcolvin/watchfiles/issues/177, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA62GGJI2IEIQY2PM774FZLVV3BYTANCNFSM54S7D3IA . You are receiving this because you are subscribed to this thread.Message ID: @.***>
Hey all- wanted to report that I can reproduce this under certain circumstances on 0.18:
$ python -c 'import platform; print(platform.platform()); print(platform.version())'
macOS-12.6.1-x86_64-i386-64bit
Darwin Kernel Version 21.6.0: Thu Sep 29 20:12:57 PDT 2022; root:xnu-8020.240.7~1/RELEASE_X86_64
$ pip show watchfiles
Name: watchfiles
Version: 0.18.1
...
#!/usr/bin/env python3
import asyncio, watchfiles
async def watch():
async for changes in watchfiles.awatch("."):
for change in changes:
print(change)
async def main():
asyncio.create_task(watch())
await asyncio.sleep(1)
asyncio.run(main())
unhandled exception during asyncio.run() shutdown
task: <Task finished name='Task-2' coro=<watch() done, defined at ./foo.py:6> exception=RuntimeError('Already borrowed')>
Traceback (most recent call last):
File "./foo.py", line 7, in watch
async for changes in watchfiles.awatch("."):
File "/lib/python3.9/site-packages/watchfiles/main.py", line 266, in awatch
yield changes
RuntimeError: Already borrowed
It seems to only happen when the coroutine using awatch happens in a task; when I asyncio.run(watch()) directly, it responds normally to SIGINT. It doesn't seem to make a difference whether the task is ignored or cancelled and awaited at the end of main():
#!/usr/bin/env python3
import asyncio, watchfiles
async def watch():
async for changes in watchfiles.awatch("."):
for change in changes:
print(change)
async def main():
t = asyncio.create_task(watch())
await asyncio.sleep(1)
t.cancel()
await t
asyncio.run(main())
Traceback (most recent call last):
File "./foo.py", line 20, in <module>
asyncio.run(main())
File "/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
return future.result()
File "./foo.py", line 16, in main
await t
File "./foo.py", line 7, in watch
async for changes in watchfiles.awatch("."):
File "/lib/python3.9/site-packages/watchfiles/main.py", line 266, in awatch
yield changes
RuntimeError: Already borrowed
Please let me know if I should move this over to the PyO3 thread!
Thanks so much, to save me the effort, it would be great if you can follow the instructions on the rust thread, and report the output there.