[bug]: Classes which inherit from EventBase have no get() method?

[d8ahazard]

Is there an existing issue for this problem?

• [X] I have searched the existing issues

Operating system

Linux

GPU vendor

Nvidia (CUDA)

GPU model

Multiple

GPU VRAM

Various

Version number

4.2.6.post1

Browser

Chrome

Python dependencies

No response

What happened

When I generate anything, I see multiple AttributeNotFound exceptions related to all classes used during generation which inherit from EventBase.

This stems from the file here:

https://github.com/invoke-ai/InvokeAI/blob/main/invokeai/app/services/events/events_common.py

And can be fixed by simply declaring a get method for EventBase like so:

def get(self, key: str, default=None):
    return self.model_dump().get(key,default)

Without this, I see no preview while generating, and no output image until I reload the page.

LOGS:

Task exception was never retrieved
future: <Task finished name='Task-139' coro=<_dispatch_as_task.<locals>.task() done, defined at /home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/dispatcher.py:51> exception=AttributeError("'BatchEnqueuedEvent' object has no attribute 'get'")>
Traceback (most recent call last):
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/dispatcher.py", line 52, in task
   await asyncio.gather(*[handler.handle((event_name, payload)) for handler in handlers])
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/handlers/local.py", line 195, in handle
   with create_span_for_handle_fn(
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/otel/utils.py", line 43, in create_span_for_handle_fn
   remote_ctx = propagate.extract(payload)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/propagate/__init__.py", line 101, in extract
   return get_global_textmap().extract(carrier, context, getter=getter)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/propagators/composite.py", line 52, in extract
   context = propagator.extract(carrier, context, getter=getter)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/trace/propagation/tracecontext.py", line 49, in extract
   header = getter.get(carrier, self._TRACEPARENT_HEADER_NAME)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/propagators/textmap.py", line 87, in get
   val = carrier.get(key, None)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/pydantic/main.py", line 811, in __getattr__
   raise AttributeError(f'{type(self).__name__!r} object has no attribute {item!r}')
AttributeError: 'BatchEnqueuedEvent' object has no attribute 'get'
[2024-07-24 19:31:07,318]::[uvicorn.access]::INFO --> 70.92.243.202:57044 - "GET /api/v1/queue/default/status HTTP/1.1" 200
[2024-07-24 19:31:07,522]::[uvicorn.access]::INFO --> 70.92.243.202:57043 - "GET /api/v1/queue/default/list HTTP/1.1" 200
Task exception was never retrieved
future: <Task finished name='Task-143' coro=<_dispatch_as_task.<locals>.task() done, defined at /home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/dispatcher.py:51> exception=AttributeError("'QueueItemStatusChangedEvent' object has no attribute 'get'")>
Traceback (most recent call last):
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/dispatcher.py", line 52, in task
   await asyncio.gather(*[handler.handle((event_name, payload)) for handler in handlers])
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/handlers/local.py", line 195, in handle
   with create_span_for_handle_fn(
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/otel/utils.py", line 43, in create_span_for_handle_fn
   remote_ctx = propagate.extract(payload)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/propagate/__init__.py", line 101, in extract
   return get_global_textmap().extract(carrier, context, getter=getter)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/propagators/composite.py", line 52, in extract
   context = propagator.extract(carrier, context, getter=getter)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/trace/propagation/tracecontext.py", line 49, in extract
   header = getter.get(carrier, self._TRACEPARENT_HEADER_NAME)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/propagators/textmap.py", line 87, in get
   val = carrier.get(key, None)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/pydantic/main.py", line 811, in __getattr__
   raise AttributeError(f'{type(self).__name__!r} object has no attribute {item!r}')
AttributeError: 'QueueItemStatusChangedEvent' object has no attribute 'get'
Task exception was never retrieved
future: <Task finished name='Task-144' coro=<_dispatch_as_task.<locals>.task() done, defined at /home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/dispatcher.py:51> exception=AttributeError("'InvocationStartedEvent' object has no attribute 'get'")>
Traceback (most recent call last):
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/dispatcher.py", line 52, in task
   await asyncio.gather(*[handler.handle((event_name, payload)) for handler in handlers])
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/handlers/local.py", line 195, in handle
   with create_span_for_handle_fn(
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/otel/utils.py", line 43, in create_span_for_handle_fn
   remote_ctx = propagate.extract(payload)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/propagate/__init__.py", line 101, in extract
   return get_global_textmap().extract(carrier, context, getter=getter)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/propagators/composite.py", line 52, in extract
   context = propagator.extract(carrier, context, getter=getter)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/trace/propagation/tracecontext.py", line 49, in extract
   header = getter.get(carrier, self._TRACEPARENT_HEADER_NAME)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/propagators/textmap.py", line 87, in get
   val = carrier.get(key, None)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/pydantic/main.py", line 811, in __getattr__
   raise AttributeError(f'{type(self).__name__!r} object has no attribute {item!r}')
AttributeError: 'InvocationStartedEvent' object has no attribute 'get'
Task exception was never retrieved
future: <Task finished name='Task-145' coro=<_dispatch_as_task.<locals>.task() done, defined at /home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/dispatcher.py:51> exception=AttributeError("'InvocationCompleteEvent' object has no attribute 'get'")>
Traceback (most recent call last):
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/dispatcher.py", line 52, in task
   await asyncio.gather(*[handler.handle((event_name, payload)) for handler in handlers])
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/handlers/local.py", line 195, in handle
   with create_span_for_handle_fn(
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/otel/utils.py", line 43, in create_span_for_handle_fn
   remote_ctx = propagate.extract(payload)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/propagate/__init__.py", line 101, in extract
   return get_global_textmap().extract(carrier, context, getter=getter)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/propagators/composite.py", line 52, in extract
   context = propagator.extract(carrier, context, getter=getter)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/trace/propagation/tracecontext.py", line 49, in extract
   header = getter.get(carrier, self._TRACEPARENT_HEADER_NAME)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/propagators/textmap.py", line 87, in get
   val = carrier.get(key, None)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/pydantic/main.py", line 811, in __getattr__
   raise AttributeError(f'{type(self).__name__!r} object has no attribute {item!r}')
AttributeError: 'InvocationCompleteEvent' object has no attribute 'get'
Task exception was never retrieved
future: <Task finished name='Task-146' coro=<_dispatch_as_task.<locals>.task() done, defined at /home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/dispatcher.py:51> exception=AttributeError("'InvocationStartedEvent' object has no attribute 'get'")>
Traceback (most recent call last):
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/dispatcher.py", line 52, in task
   await asyncio.gather(*[handler.handle((event_name, payload)) for handler in handlers])
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/handlers/local.py", line 195, in handle
   with create_span_for_handle_fn(
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/otel/utils.py", line 43, in create_span_for_handle_fn
   remote_ctx = propagate.extract(payload)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/propagate/__init__.py", line 101, in extract
   return get_global_textmap().extract(carrier, context, getter=getter)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/propagators/composite.py", line 52, in extract
   context = propagator.extract(carrier, context, getter=getter)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/trace/propagation/tracecontext.py", line 49, in extract
   header = getter.get(carrier, self._TRACEPARENT_HEADER_NAME)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/propagators/textmap.py", line 87, in get
   val = carrier.get(key, None)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/pydantic/main.py", line 811, in __getattr__
   raise AttributeError(f'{type(self).__name__!r} object has no attribute {item!r}')
AttributeError: 'InvocationStartedEvent' object has no attribute 'get'
Task exception was never retrieved
future: <Task finished name='Task-147' coro=<_dispatch_as_task.<locals>.task() done, defined at /home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/dispatcher.py:51> exception=AttributeError("'ModelLoadStartedEvent' object has no attribute 'get'")>
Traceback (most recent call last):
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/dispatcher.py", line 52, in task
   await asyncio.gather(*[handler.handle((event_name, payload)) for handler in handlers])
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/handlers/local.py", line 195, in handle
   with create_span_for_handle_fn(
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/fastapi_events/otel/utils.py", line 43, in create_span_for_handle_fn
   remote_ctx = propagate.extract(payload)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/propagate/__init__.py", line 101, in extract
   return get_global_textmap().extract(carrier, context, getter=getter)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/propagators/composite.py", line 52, in extract
   context = propagator.extract(carrier, context, getter=getter)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/trace/propagation/tracecontext.py", line 49, in extract
   header = getter.get(carrier, self._TRACEPARENT_HEADER_NAME)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/opentelemetry/propagators/textmap.py", line 87, in get
   val = carrier.get(key, None)
 File "/home/d8/miniconda3/envs/INVK/lib/python3.10/site-packages/pydantic/main.py", line 811, in __getattr__
   raise AttributeError(f'{type(self).__name__!r} object has no attribute {item!r}')
AttributeError: 'ModelLoadStartedEvent' object has no attribute 'get'
[2024-07-24 19:31:07,996]::[uvicorn.access]::INFO --> 70.92.243.202:57043 - "GET /assets/images/invoke-alert-favicon.svg HTTP/1.1" 200
Task exception was never retrieved

What you expected to happen

Preview whle generating, an output image, etc.

How to reproduce the problem

I have no idea why this is happening with this issue and nobody else reported it.

I literally just installed the latest version + xformers from pypi:

"pip install InvokeAI[xformers] --use-pep517 --extra-index-url https://download.pytorch.org/whl/cu121 --upgrade"

Additional context

No response

Discord username

No response

[psychedelicious]

Guessing you have the opentelemetry package installed in that python env. This triggers some additional behaviour from fastapi-events. See #6644.

If you can't remove opentelemetry, the issue might be fixed if you update to v4.2.7rc1.

[ebr]

The fix (possibly not the ideal fix) is in this PR into fastapi-events: https://github.com/melvinkcx/fastapi-events/pull/64. Please feel free to upvote / comment to bring it to the maintainer's attention.

@psychedelicious any thoughts on it?

[d8ahazard]

The fix (possibly not the ideal fix) is in this PR into fastapi-events: melvinkcx/fastapi-events#64. Please feel free to upvote / comment to bring it to the maintainer's attention.

@psychedelicious any thoughts on it?

IMHO, were this my project, I'd probably add the fix I suggested regardless? Then, even if it gets fixed upstream down the road, the get() function should have no impact. ;)

[psychedelicious]

I don't think adding a get method to the pydantic model is an appropriate solution. We've made contributions to the upstream library in the past, I'm sure the maintainer will either accept the PR or fix it in a different way.

Did you check if the opentelemetry package is installed in your python env?

[d8ahazard]

I don't think adding a get method to the pydantic model is an appropriate solution. We've made contributions to the upstream library in the past, I'm sure the maintainer will either accept the PR or fix it in a different way.

Did you check if the opentelemetry package is installed in your python env?

I did, and it is. Seems it's being installed by the invokeai package on pypi.

I have my temporary patch for now, I can wait until the next package is released. Thanks. :D

[psychedelicious]

Huh, that's unexpected - I don't have it in my install. Is it possible it was installed by some other means? Definitely shouldn't be in the pypi package!

[d8ahazard]

Aaaaah. I know. I have a base environment set up in Conda that I use for the various apps we run. I bet it's part of the default packages I install.

I'll make sure to add a custom bit to my installer to remove it.

[psychedelicious]

Ah ok cool, sounds like that's the culprit. We'll keep this issue open until the root issue is resolved.

[d8ahazard]

@psychedelicious - FWIW, removing opentelemetry did not fix the issue, nor did the latest build (4.2.7). I still had to manually apply my patch.

[psychedelicious]

@d8ahazard just to confirm, after removing the otel package you had the same error (which references the otel pkg)? That doesn't sound possible if it was removed fully.