OS X: Shm issue `OSError: [Errno 63] File name too long:`

[torresnelson]

trafficstars

Using this command piker -l info -b binance chart btcusdt.binanc I'm getting this error:

Apr 18 10:50:19 (brokerd.binance, 15316, piker.data.feed.allocate_persistent_feed)) [WARNING] piker.data.feed feed.py:383 btcusdt.binance feed task terminated
Apr 18 10:50:19 (brokerd.binance, 15316, piker.brokers.binance.stream_quotes)) [INFO] tractor.trionics._mngrs _mngrs.py:215 De-allocating resource for (4934515248, ())
Remote context error for ('brokerd.binance', '0de12a6f-0628-4929-b437-6ba6f3b4632b'):c2015eba-ca1f-4225-bbf2-82e957ccadb8:
Traceback (most recent call last):
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/src/tractor/tractor/_actor.py", line 224, in _invoke
    raise ContextCancelled(
tractor._exceptions.ContextCancelled: _setup_persistent_brokerd was remotely cancelled by its caller ('pikerd', 'a82b89dc-0af0-4bbf-a18d-cedf6b129fa1')

Nursery for ('pikerd', 'a82b89dc-0af0-4bbf-a18d-cedf6b129fa1') errored with [Errno 63] File name too long: '/btcusdt.binance.fsp.dolla_vlm.pikerd.a82b89dc-0af0-4bbf-a18d-cedf6b129fa1',
Traceback (most recent call last):
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/src/tractor/tractor/trionics/_mngrs.py", line 179, in maybe_open_context
    value = _Cache.values[ctx_key]
KeyError: (5177244352, ())

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/src/tractor/tractor/_supervise.py", line 327, in _open_and_supervise_one_cancels_all_nursery
    yield anursery
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/src/tractor/tractor/_supervise.py", line 494, in open_nursery
    yield anursery
  File "/Users/ntorres/Documents/piker/piker/_daemon.py", line 178, in open_pikerd
    yield _services
  File "/Users/ntorres/Documents/piker/piker/_daemon.py", line 279, in maybe_open_pikerd
    yield None
  File "/Users/ntorres/Documents/piker/piker/ui/_exec.py", line 175, in main
    await func(*((instance,) + args))
  File "/Users/ntorres/Documents/piker/piker/ui/_app.py", line 100, in _async_main
    async with (
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/lib/python3.10/site-packages/trio/_core/_run.py", line 813, in __aexit__
    raise combined_error_from_nursery
  File "/Users/ntorres/Documents/piker/piker/ui/_display.py", line 549, in display_symbol_data
    async with trio.open_nursery() as ln:
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/lib/python3.10/site-packages/trio/_core/_run.py", line 813, in __aexit__
    raise combined_error_from_nursery
  File "/Users/ntorres/Documents/piker/piker/ui/_fsp.py", line 687, in open_vlm_displays
    dvlm_shm, started = await admin.start_engine_task(
  File "/Users/ntorres/Documents/piker/piker/ui/_fsp.py", line 457, in start_engine_task
    key, dst_shm, opened = maybe_mk_fsp_shm(
  File "/Users/ntorres/Documents/piker/piker/fsp/_api.py", line 193, in maybe_mk_fsp_shm
    shm, opened = maybe_open_shm_array(
  File "/Users/ntorres/Documents/piker/piker/data/_sharedmem.py", line 553, in maybe_open_shm_array
    return attach_shm_array(token=token, **kwargs), False
  File "/Users/ntorres/Documents/piker/piker/data/_sharedmem.py", line 474, in attach_shm_array
    shm = SharedMemory(name=key)
  File "/Users/ntorres/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/shared_memory.py", line 103, in __init__
    self._fd = _posixshmem.shm_open(
OSError: [Errno 63] File name too long: '/btcusdt.binance.fsp.dolla_vlm.pikerd.a82b89dc-0af0-4bbf-a18d-cedf6b129fa1'
Apr 18 10:50:19 (brokerd.binance, 15316, tractor._actor.Actor._async_main)) [INFO] tractor _actor.py:1209 Runtime nursery complete
Apr 18 10:50:19 (brokerd.binance, 15316, tractor._actor.Actor._async_main)) [INFO] tractor _actor.py:1213 Closing all actor lifetime contexts
Apr 18 10:50:19 (brokerd.binance, 15316, no task context)) [INFO] tractor._entry _entry.py:123 Actor ('brokerd.binance', '0de12a6f-0628-4929-b437-6ba6f3b4632b') terminated
Cancelling existing result waiter task for ('brokerd.binance', '0de12a6f-0628-4929-b437-6ba6f3b4632b')
Root actor crashed:
Traceback (most recent call last):
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/src/tractor/tractor/trionics/_mngrs.py", line 179, in maybe_open_context
    value = _Cache.values[ctx_key]
KeyError: (5177244352, ())

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/src/tractor/tractor/_root.py", line 204, in open_root_actor
    yield actor
  File "/Users/ntorres/Documents/piker/piker/_daemon.py", line 178, in open_pikerd
    yield _services
  File "/Users/ntorres/Documents/piker/piker/_daemon.py", line 279, in maybe_open_pikerd
    yield None
  File "/Users/ntorres/Documents/piker/piker/ui/_exec.py", line 175, in main
    await func(*((instance,) + args))
  File "/Users/ntorres/Documents/piker/piker/ui/_app.py", line 100, in _async_main
    async with (
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/lib/python3.10/site-packages/trio/_core/_run.py", line 813, in __aexit__
    raise combined_error_from_nursery
  File "/Users/ntorres/Documents/piker/piker/ui/_display.py", line 549, in display_symbol_data
    async with trio.open_nursery() as ln:
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/lib/python3.10/site-packages/trio/_core/_run.py", line 813, in __aexit__
    raise combined_error_from_nursery
  File "/Users/ntorres/Documents/piker/piker/ui/_fsp.py", line 687, in open_vlm_displays
    dvlm_shm, started = await admin.start_engine_task(
  File "/Users/ntorres/Documents/piker/piker/ui/_fsp.py", line 457, in start_engine_task
    key, dst_shm, opened = maybe_mk_fsp_shm(
  File "/Users/ntorres/Documents/piker/piker/fsp/_api.py", line 193, in maybe_mk_fsp_shm
    shm, opened = maybe_open_shm_array(
  File "/Users/ntorres/Documents/piker/piker/data/_sharedmem.py", line 553, in maybe_open_shm_array
    return attach_shm_array(token=token, **kwargs), False
  File "/Users/ntorres/Documents/piker/piker/data/_sharedmem.py", line 474, in attach_shm_array
    shm = SharedMemory(name=key)
  File "/Users/ntorres/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/shared_memory.py", line 103, in __init__
    self._fd = _posixshmem.shm_open(
OSError: [Errno 63] File name too long: '/btcusdt.binance.fsp.dolla_vlm.pikerd.a82b89dc-0af0-4bbf-a18d-cedf6b129fa1'
Traceback (most recent call last):
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/src/tractor/tractor/trionics/_mngrs.py", line 179, in maybe_open_context
    value = _Cache.values[ctx_key]
KeyError: (5177244352, ())

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ntorres/Documents/piker/piker/ui/_exec.py", line 175, in main
    await func(*((instance,) + args))
  File "/Users/ntorres/Documents/piker/piker/ui/_app.py", line 100, in _async_main
    async with (
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/lib/python3.10/site-packages/trio/_core/_run.py", line 813, in __aexit__
    raise combined_error_from_nursery
  File "/Users/ntorres/Documents/piker/piker/ui/_display.py", line 549, in display_symbol_data
    async with trio.open_nursery() as ln:
  File "/Users/ntorres/.pyenv/versions/3.10.4/envs/pikers/lib/python3.10/site-packages/trio/_core/_run.py", line 813, in __aexit__
    raise combined_error_from_nursery
  File "/Users/ntorres/Documents/piker/piker/ui/_fsp.py", line 687, in open_vlm_displays
    dvlm_shm, started = await admin.start_engine_task(
  File "/Users/ntorres/Documents/piker/piker/ui/_fsp.py", line 457, in start_engine_task
    key, dst_shm, opened = maybe_mk_fsp_shm(
  File "/Users/ntorres/Documents/piker/piker/fsp/_api.py", line 193, in maybe_mk_fsp_shm
    shm, opened = maybe_open_shm_array(
  File "/Users/ntorres/Documents/piker/piker/data/_sharedmem.py", line 553, in maybe_open_shm_array
    return attach_shm_array(token=token, **kwargs), False
  File "/Users/ntorres/Documents/piker/piker/data/_sharedmem.py", line 474, in attach_shm_array
    shm = SharedMemory(name=key)
  File "/Users/ntorres/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/shared_memory.py", line 103, in __init__
    self._fd = _posixshmem.shm_open(
OSError: [Errno 63] File name too long: '/btcusdt.binance.fsp.dolla_vlm.pikerd.a82b89dc-0af0-4bbf-a18d-cedf6b129fa1'
< Exiting ui._fsp.start_fsp_displays, total time: 4649.5217 ms

[goodboy]

OSError: [Errno 63] File name too long: '/btcusdt.binance.fsp.dolla_vlm.pikerd.a82b89dc-0af0-4bbf-a18d-cedf6b129fa1'

Very interesting. I guess we could try truncating it on OS X? I'm not sure about how peeps normally solve this issue tho i have definitely seen it before when working such machines.

this is actually something that needs to get dug into anyway - better organizing shm files on the system to be,

1. easier to wipe when things go wrong and files are left stale
2. clearer to the user which array / entry maps to which app/service.

[torresnelson]

Truncating it at mk_fsp_shm_key() throws the same error: OSError: [Errno 63] File name too long: '/btcusdt.binance.fsp.dolla_vlm.pikerd.c84'

[goodboy]

I've also seen a new issue with "too many open files" as an OSError coming from this code.

We'll need to dig in at some point but for this simple case what is the max "file name length" on OS X?

[goodboy]

@torresnelson yeah not sure what to say (btw) bc i don't run OS X, so you'll have to do some tinkering on your own or write a test we can stick in CI.