aim icon indicating copy to clipboard operation
aim copied to clipboard
verified publisher icon aimhubio

aim runs cp on fresh aim init causes KeyError: No key ('contexts', ...) is present

Open YodaEmbedding opened this issue 3 years ago • 4 comments

🐛 Bug

Aim Runs page is blank and raises error after running aim runs cp on fresh aim init.

To reproduce

mkdir new_repo && cd new_repo
aim init
cd ../old_repo
aim runs cp $(aim runs ls | head -n -1) --destination=../new_repo
cd ../new_repo
aim up

Navigate to Runs page.

Output:

...
  File ".../aim/web/api/runs/utils.py", line 251, in run_search_result_streamer
    'traces': run.collect_sequence_info(sequence_types='metric'),
  File ".../aim/sdk/run.py", line 669, in collect_sequence_info
    ctx_dict = self.idx_to_ctx(idx).to_dict()
  File ".../aim/sdk/run.py", line 353, in idx_to_ctx
    return self._tracker.idx_to_ctx(idx)
  File ".../aim/sdk/tracker.py", line 80, in idx_to_ctx
    ctx = Context(self.meta_tree['contexts', idx])
  File "aim/storage/treeview.py", line 51, in aim.storage.treeview.TreeView.__getitem__
  File "aim/storage/containertreeview.py", line 74, in aim.storage.containertreeview.ContainerTreeView.collect
KeyError: "No key ('contexts', 917550741257467117) is present."

Expected behavior

No error. Non-blank runs page.

Environment

  • Aim Version: 3.13.2
  • Python version: 3.8
  • pip version: 22.1
  • OS: Linux
  • Any other relevant information

Additional context

N/A

YodaEmbedding avatar Sep 21 '22 07:09 YodaEmbedding

@YodaEmbedding thanks for reporting. Looking into it!

gorarakelyan avatar Sep 21 '22 10:09 gorarakelyan

could be related to the following comment: https://github.com/aimhubio/aim/issues/1246#issuecomment-1253969553

alberttorosyan avatar Sep 22 '22 10:09 alberttorosyan

I get the same issue

KeyError: "No key ('contexts', ...) is present."

when copying runs from a 3.11.2 instance to another machine which is also running aim 3.11.2

Similar behavior on 3.13.3.

Pyrestone avatar Sep 22 '22 19:09 Pyrestone

Just wanted to say that I'm seeing this on 3.16.0 as well. Everything works correctly on the original repository I copied from. Additionally, here's a fun exception that gets thrown on the command line when I navigate to the Runs page.

click to show exception info 
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/fastapi/applications.py", line 270, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/starlette/routing.py", line 706, in __call__
    await route.handle(scope, receive, send)
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/starlette/routing.py", line 443, in handle
    await self.app(scope, receive, send)
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/fastapi/applications.py", line 270, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/starlette/middleware/gzip.py", line 24, in __call__
    await responder(scope, receive, send)
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/starlette/middleware/gzip.py", line 44, in __call__
    await self.app(scope, receive, self.send_with_gzip)
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/starlette/routing.py", line 706, in __call__
    await route.handle(scope, receive, send)
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/starlette/routing.py", line 69, in app
    await response(scope, receive, send)
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/starlette/responses.py", line 266, in __call__
    async with anyio.create_task_group() as task_group:
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
    raise exceptions[0]
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/starlette/responses.py", line 269, in wrap
    await func()
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/starlette/responses.py", line 258, in stream_response
    async for chunk in self.body_iterator:
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/aim/web/api/runs/utils.py", line 259, in run_search_result_streamer
    run_dict[run.hash]['traces'] = run.collect_sequence_info(sequence_types='metric')
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/aim/sdk/run.py", line 661, in collect_sequence_info
    ctx_dict = self.idx_to_ctx(idx).to_dict()
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/aim/sdk/run.py", line 335, in idx_to_ctx
    return self._tracker.idx_to_ctx(idx)
  File "/Users/REDACTED/venv/lib/python3.10/site-packages/aim/sdk/tracker.py", line 80, in idx_to_ctx
    ctx = Context(self.meta_tree['contexts', idx])
  File "aim/storage/treeview.py", line 51, in aim.storage.treeview.TreeView.__getitem__
  File "aim/storage/containertreeview.py", line 73, in aim.storage.containertreeview.ContainerTreeView.collect
KeyError: "No key ('contexts', -5834889044196056041) is present."

csparker247 avatar Feb 08 '23 01:02 csparker247