mitogen icon indicating copy to clipboard operation
mitogen copied to clipboard
verified publisher icon mitogen-hq

BlockingIOError: [Errno 35] Resource temporarily unavailable

Open moreati opened this issue 1 year ago • 8 comments

Seen in CI on macOS 13 https://github.com/mitogen-hq/mitogen/actions/runs/11669080981/job/32490230504?pr=1176

TASK [Create /tmp/512mb.zero creates=/tmp/512mb.zero, _raw_params=dd if=/dev/zero of=/tmp/512mb.zero bs=1048576 count=512;
chmod go= /tmp/512mb.zero
] ***
Monday 04 November 2024  17:08:28 +0000 (0:00:00.007)       0:00:26.162 ******* 
changed: [target]
TASK [Fetch /tmp/512mb.zero src=/tmp/512mb.zero, dest=/tmp/fetch-{{ inventory_hostname }}-512mb.zero, flat=True] ***
Monday 04 November 2024  17:08:30 +0000 (0:00:01.338)       0:00:27.501 ******* 
ERROR! [mux  5271] 17:08:37.240824 E mitogen: <Stream unix_client.5683 #1dd0> crashed
Traceback (most recent call last):
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 3677, in _call
    func(self)
    ~~~~^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 1923, in on_transmit
    self.protocol.on_transmit(broker)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2371, in on_transmit
    self._writer.on_transmit(broker)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2111, in on_transmit
    written = self._protocol.stream.transmit_side.write(buf)
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2237, in write
    written, disconnected = io_op(os.write, self.fd, s)
                            ~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 595, in io_op
    return func(*args), None
           ~~~~^^^^^^^
BlockingIOError: [Errno 35] Resource temporarily unavailable

Originally posted by @moreati in https://github.com/mitogen-hq/mitogen/issues/1133#issuecomment-2455377609

moreati avatar Nov 06 '24 09:11 moreati

Again https://github.com/mitogen-hq/mitogen/actions/runs/11700584431/job/32584889627

PLAY [regression/issue_615_streaming_transfer.yml] *****************************
TASK [include_tasks _raw_params=_mitogen_only.yml] *****************************
Wednesday 06 November 2024  09:20:58 +0000 (0:00:00.120)       0:00:26.685 **** 
included: /Users/runner/work/mitogen/mitogen/tests/ansible/regression/_mitogen_only.yml for target
TASK [meta _raw_params=end_play] ***********************************************
Wednesday 06 November 2024  09:20:58 +0000 (0:00:00.039)       0:00:26.725 **** 
skipping: [target]
TASK [Create /tmp/512mb.zero creates=/tmp/512mb.zero, _raw_params=dd if=/dev/zero of=/tmp/512mb.zero bs=1048576 count=512;
chmod go= /tmp/512mb.zero
] ***
Wednesday 06 November 2024  09:20:58 +0000 (0:00:00.014)       0:00:26.739 **** 
changed: [target]
TASK [Fetch /tmp/512mb.zero src=/tmp/512mb.zero, dest=/tmp/fetch-{{ inventory_hostname }}-512mb.zero, flat=True] ***
Wednesday 06 November 2024  09:21:00 +0000 (0:00:01.685)       0:00:28.424 **** 
ERROR! [mux  5601] 09:21:05.103152 E mitogen: <Stream unix_client.5949 #8cd0> crashed
Traceback (most recent call last):
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 3677, in _call
    func(self)
    ~~~~^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 1923, in on_transmit
    self.protocol.on_transmit(broker)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2371, in on_transmit
    self._writer.on_transmit(broker)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2111, in on_transmit
    written = self._protocol.stream.transmit_side.write(buf)
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2237, in write
    written, disconnected = io_op(os.write, self.fd, s)
                            ~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 595, in io_op
    return func(*args), None
           ~~~~^^^^^^^
BlockingIOError: [Errno 35] Resource temporarily unavailable

moreati avatar Nov 06 '24 09:11 moreati

Again, macos (Loc_313_10, py313-mode_localhost-ansible10)

PLAY [regression/issue_615_streaming_transfer.yml] *****************************
TASK [include_tasks _raw_params=_mitogen_only.yml] *****************************
Thursday 07 November 2024  14:22:43 +0000 (0:00:00.113)       0:00:25.291 ***** 
included: /Users/runner/work/mitogen/mitogen/tests/ansible/regression/_mitogen_only.yml for target
TASK [meta _raw_params=end_play] ***********************************************
Thursday 07 November 2024  14:22:43 +0000 (0:00:00.037)       0:00:25.328 ***** 
skipping: [target]
TASK [Create /tmp/512mb.zero creates=/tmp/512mb.zero, _raw_params=dd if=/dev/zero of=/tmp/512mb.zero bs=1048576 count=512;
chmod go= /tmp/512mb.zero
] ***
Thursday 07 November 2024  14:22:43 +0000 (0:00:00.010)       0:00:25.339 ***** 
changed: [target]
TASK [Fetch /tmp/512mb.zero src=/tmp/512mb.zero, dest=/tmp/fetch-{{ inventory_hostname }}-512mb.zero, flat=True] ***
Thursday 07 November 2024  14:22:44 +0000 (0:00:01.361)       0:00:26.700 ***** 
ERROR! [mux  3519] 14:22:49.843910 E mitogen: <Stream unix_client.3873 #db50> crashed
Traceback (most recent call last):
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 3677, in _call
    func(self)
    ~~~~^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 1923, in on_transmit
    self.protocol.on_transmit(broker)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2371, in on_transmit
    self._writer.on_transmit(broker)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2111, in on_transmit
    written = self._protocol.stream.transmit_side.write(buf)
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2237, in write
    written, disconnected = io_op(os.write, self.fd, s)
                            ~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 595, in io_op
    return func(*args), None
           ~~~~^^^^^^^
BlockingIOError: [Errno 35] Resource temporarily unavailable

moreati avatar Nov 07 '24 14:11 moreati

https://github.com/mitogen-hq/mitogen/pull/1194 has hit this 3 times in a row

moreati avatar Nov 07 '24 17:11 moreati

#1177 twice in a row (job 394, 2 attempts), once it occurs the build hangs until it is concelled or times out (currently 2 hours)

TASK [Create /tmp/512mb.zero creates=/tmp/512mb.zero, _raw_params=dd if=/dev/zero of=/tmp/512mb.zero bs=1048576 count=512;
chmod go= /tmp/512mb.zero
] ***
Wednesday 27 November 2024  10:45:54 +0000 (0:00:00.011)       0:00:28.576 **** 
changed: [target]

TASK [Fetch /tmp/512mb.zero src=/tmp/512mb.zero, dest=/tmp/fetch-{{ inventory_hostname }}-512mb.zero, flat=True] ***
Wednesday 27 November 2024  10:45:55 +0000 (0:00:01.644)       0:00:30.220 **** 
ERROR! [mux  6790] 10:46:04.059200 E mitogen: <Stream unix_client.7491 #c550> crashed
Traceback (most recent call last):
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 3677, in _call
    func(self)
    ~~~~^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 1923, in on_transmit
    self.protocol.on_transmit(broker)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2371, in on_transmit
    self._writer.on_transmit(broker)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2111, in on_transmit
    written = self._protocol.stream.transmit_side.write(buf)
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2237, in write
    written, disconnected = io_op(os.write, self.fd, s)
                            ~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 595, in io_op
    return func(*args), None
           ~~~~^^^^^^^
BlockingIOError: [Errno 35] Resource temporarily unavailable

moreati avatar Nov 27 '24 11:11 moreati

This stopped happening in CI on Nov 29. I now cannot reproduce

moreati avatar Dec 02 '24 09:12 moreati

Spoke too soon, It's back https://github.com/mitogen-hq/mitogen/actions/runs/12116635990/job/33777406222

moreati avatar Dec 02 '24 10:12 moreati

Judging by https://github.com/mitogen-hq/mitogen/actions/runs/12117305356/job/33779477126 I don't think there's a pattern wrt to Ansible versions or Python versions. So it's probably something non-deterministic such as a race condition.

moreati avatar Dec 02 '24 10:12 moreati

It's baaaaaack https://github.com/mitogen-hq/mitogen/actions/runs/12237211642/job/34139694093?pr=1200

moreati avatar Dec 09 '24 16:12 moreati

Possibly related to tickets #680, #712, #745, & #749 - they mention "errno 11", which is EAGAIN/EWOULDBLOCK on Linux.

moreati avatar Jul 06 '25 21:07 moreati

Again https://github.com/mitogen-hq/mitogen/actions/runs/18906830627/job/53967000783

TASK [Fetch /tmp/512mb.zero src=/tmp/512mb.zero, dest=/tmp/fetch-{{ inventory_hostname }}-512mb.zero, flat=True] ***
Wednesday 29 October 2025  11:51:41 +0000 (0:00:04.122)       0:00:36.706 ***** 
Error: : [mux  7705] 11:51:48.700045 E mitogen: <Stream unix_client.8796 #1450> crashed
Traceback (most recent call last):
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 3685, in _call
    func(self)
    ~~~~^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 1934, in on_transmit
    self.protocol.on_transmit(broker)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2377, in on_transmit
    self._writer.on_transmit(broker)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2122, in on_transmit
    written = self._protocol.stream.transmit_side.write(buf)
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2243, in write
    written, disconnected = io_op(os.write, self.fd, s)
                            ~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 596, in io_op
    return func(*args), None
           ~~~~^^^^^^^
BlockingIOError: [Errno 35] Resource temporarily unavailable

moreati avatar Oct 29 '25 12:10 moreati