tiled
tiled copied to clipboard
bluesky
Tiled CLI register needs container to exist to register objects with a prefix
Use case:
Have a datasets organized by subdirectories in /storage/. Each dataset is a subdir ie /storage/dataset1. I would like to register sub-directories individually like (as new datasets arrive):
tiled register http://127.0.0.1:8000/ /storage/dataset1/ --prefix=/dataset1
This fails because the Container /dataset1 does not exist.
There is no current way to create the container via the CLI (currently have to use create_container in python API)
From discussion with @dylanmcreynolds
Here is the error that I get. It looks like key_from_filename is None during some part of the registration when --prefix is set. See tiled/client/register.py:166
Interestingly settings appears to have a non-None key_from_filename function. I think that this is bug and key_from_filename should be pulled from settings
# tiled register http://localhost:8000/ /mf-storage-prod/034bqac92h0kz0r89cdd330jv0/ --api-key=XXX --prefix=/034bqac92h0kz0r89cdd330jv0
...
│ │
│ /opt/venv/lib/python3.12/site-packages/tiled/client/register.py:166 in register │
│ │
│ 163 │ for segment in prefix_parts: │
│ 164 │ │ child_node = await anyio.to_thread.run_sync(node.get, segment) │
│ 165 │ │ if child_node is None: │
│ ❱ 166 │ │ │ key = key_from_filename(segment) │
│ 167 │ │ │ await create_node_or_drop_collision( │
│ 168 │ │ │ │ node, │
│ 169 │ │ │ │ structure_family=StructureFamily.container, │
│ │
│ ╭────────────────────────────────────────────────────────────── locals ──────────────────────────────────────────────────────────────╮ │
│ │ adapters_by_mimetype = {} │ │
│ │ child_node = None │ │
│ │ filter = None │ │
│ │ key_from_filename = None │ │
│ │ mimetype_detection_hook = None │ │
│ │ mimetypes_by_file_ext = {} │ │
│ │ node = <Container {'231219_171929_picam_readout'}> │ │
│ │ overwrite = True │ │
│ │ parsed_walkers = [<function group_image_sequences at 0x7fffb96822a0>, <function one_node_per_item at 0x7fffb9682160>] │ │
│ │ path = PosixPath('/mf-storage-prod/034bqac92h0kz0r89cdd330jv0') │ │
│ │ prefix = '/034bqac92h0kz0r89cdd330jv0' │ │
│ │ prefix_parts = ['034bqac92h0kz0r89cdd330jv0'] │ │
│ │ segment = '034bqac92h0kz0r89cdd330jv0' │ │
│ │ settings = Settings( │ │
│ │ │ adapters_by_mimetype=ChainMap({}, <OneShotCachedMap({'image/tiff': <lazy>, │ │
│ │ 'multipart/related;type=image/tiff': <lazy>, 'image/jpeg': <lazy>, 'multipart/related;type=image/jpeg': │ │
│ │ <lazy>, 'text/csv': <lazy>, 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet': <lazy>, │ │
│ │ 'application/x-hdf5': <lazy>, 'application/x-netcdf': <lazy>, 'application/x-parquet': <lazy>, │ │
│ │ 'application/x-parquet;structure=sparse': <lazy>, 'application/x-zarr': <lazy>, │ │
│ │ 'application/x-awkward-buffers': <lazy>, 'application/vnd.apache.arrow.file': <lazy>})>), │ │
│ │ │ mimetypes_by_file_ext=ChainMap({}, {'.h5': 'application/x-hdf5', '.nxs': 'application/x-hdf5', │ │
│ │ '.hdf': 'application/x-hdf5', '.hdf5': 'application/x-hdf5', '.csv': 'text/csv', '.zarr': │ │
│ │ 'application/x-zarr'}), │ │
│ │ │ mimetype_detection_hook=None, │ │
│ │ │ key_from_filename=<function strip_suffixes at 0x7ffffd068ae0>, │ │
│ │ │ filter=<function default_filter at 0x7fffb96819e0> │ │
│ │ ) │ │
│ │ walkers = None │ │
│ ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: 'NoneType' object is not callable
The following diff fixes this immediate problem, but returns me back to the original problem of the container not existing.
--- a/tiled/client/register.py
+++ b/tiled/client/register.py
@@ -163,7 +163,7 @@ async def register(
for segment in prefix_parts:
child_node = await anyio.to_thread.run_sync(node.get, segment)
if child_node is None:
- key = key_from_filename(segment)
+ key = settings.key_from_filename(segment)
await create_node_or_drop_collision(
node,
structure_family=StructureFamily.container,
Now the problem is container does not exist:
tiled register http://localhost:8000/ /mf-storage-prod/034bqac92h0kz0r89cdd330jv0/ --api-key=XXX --prefix=/034bqac92h0kz0r89cdd330jv0
....
│ │
│ /opt/venv/lib/python3.12/site-packages/tiled/client/utils.py:84 in handle_error │
│ │
│ 81 │ │ │ │ # instead of dumping HTML into the terminal. │
│ 82 │ │ │ │ detail = response.reason_phrase │
│ 83 │ │ │ message = f"{exc.response.status_code}: " f"{detail} " f"{exc.request.url}" │
│ ❱ 84 │ │ │ raise ClientError(message, exc.request, exc.response) from exc │
│ 85 │ │ else: │
│ 86 │ │ │ raise │
│ 87 │
│ │
│ ╭───────────────────────────────────────────── locals ─────────────────────────────────────────────╮ │
│ │ detail = "No such entry: ['034bqac92h0kz0r89cdd330jv0']" │ │
│ │ message = "404: No such entry: ['034bqac92h0kz0r89cdd330jv0'] http://localhost:8000/api/v1/"+35 │ │
│ │ response = <Response [404 Not Found]> │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ClientError: 404: No such entry: ['034bqac92h0kz0r89cdd330jv0'] http://localhost:8000/api/v1/register/034bqac92h0kz0r89cdd330jv0
Server Side logs associated:
[d6fe2a8a5cfd1281] 127.0.0.1:46388 (public) - "GET /api/v1/ HTTP/1.1" 200 OK
[4f987cee67076f0b] 127.0.0.1:46388 ('service') - "GET /api/v1/metadata/?include_data_sources=false HTTP/1.1" 200 OK
[d2e746223ebbe2cc] 127.0.0.1:46388 ('service') - "GET /api/v1/metadata/034bqac92h0kz0r89cdd330jv0 HTTP/1.1" 404 Not Found
[2dbea10da5000201] 127.0.0.1:46388 ('service') - "POST /api/v1/metadata/ HTTP/1.1" 200 OK
[a81b532ab7cb2653] 127.0.0.1:46388 ('service') - "GET /api/v1/metadata/034bqac92h0kz0r89cdd330jv0 HTTP/1.1" 200 OK
[931b1fa74f340146] 127.0.0.1:46388 ('service') - "DELETE /api/v1/nodes/034bqac92h0kz0r89cdd330jv0 HTTP/1.1" 200 OK
[913ac4e9384e68df] 127.0.0.1:46388 ('service') - "POST /api/v1/register/034bqac92h0kz0r89cdd330jv0 HTTP/1.1" 404 Not Found
[79c6c9563072345c] 127.0.0.1:46388 ('service') - "GET /api/v1/search/?page%5Boffset%5D=0&fields=&sort= HTTP/1.1" 200 OK
[32c7f71c1fed9637] 127.0.0.1:46388 ('service') - "GET /api/v1/search/034bqac92h0kz0r89cdd330jv0?page%5Boffset%5D=0&fields=&sort= HTTP/1.1" 404 Not Found
[2f933fb09d19d476] 127.0.0.1:46388 ('service') - "GET /api/v1/search/034bqac92h0kz0r89cdd330jv0?page%5Boffset%5D=0&fields=&sort= HTTP/1.1" 404 Not Found
[9802a15df9cddf4d] 127.0.0.1:46388 ('service') - "GET /api/v1/search/034bqac92h0kz0r89cdd330jv0?page%5Boffset%5D=0&fields=&sort= HTTP/1.1" 404 Not Found
[9775beeb9dd1dba0] 127.0.0.1:46388 ('service') - "GET /api/v1/search/034bqac92h0kz0r89cdd330jv0?page%5Boffset%5D=0&fields=&sort= HTTP/1.1" 404 Not Found
[e75821f5d133ba33] 127.0.0.1:46388 ('service') - "GET /api/v1/search/034bqac92h0kz0r89cdd330jv0?page%5Boffset%5D=0&fields=&sort= HTTP/1.1" 404 Not Found
[a5edec636e7166d8] 127.0.0.1:46388 ('service') - "GET /api/v1/search/034bqac92h0kz0r89cdd330jv0?page%5Boffset%5D=0&fields=&sort= HTTP/1.1" 404 Not Found
[65418d54f667db21] 127.0.0.1:46388 ('service') - "GET /api/v1/search/034bqac92h0kz0r89cdd330jv0?page%5Boffset%5D=0&fields=&sort= HTTP/1.1" 404 Not Found
[b7d4dbbdd31d4a17] 127.0.0.1:46388 ('service') - "GET /api/v1/search/034bqac92h0kz0r89cdd330jv0?page%5Boffset%5D=0&fields=&sort= HTTP/1.1" 404 Not Found