feat: image block for claude

[ntindle]

I wanted claude to take image inputs

Changes 🏗️

Adds image inputs to claude (only latest message)

Checklist 📋

For code changes:

• [x] I have clearly listed my changes in the PR description
• [x] I have made a test plan
• [x] I have tested my changes according to the test plan:
  • [x] Built agent with it

[qodo-code-review[bot]]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 Security concerns Input validation: The code processes base64 encoded image data from user input without proper validation of the content or size limits. This could potentially lead to memory issues or processing of malicious content. Consider adding validation for maximum image size and proper base64/MIME type validation.

⚡ Recommended focus areas for review Error Handling The error handling in the parse_response function returns partial results even when validation fails. Consider returning empty dict consistently on all error cases. try: parsed = json.loads(resp) if not isinstance(parsed, dict): return {}, f"Expected a dictionary, but got {type(parsed)}" if input_data.expected_format: miss_keys = set(input_data.expected_format.keys()) - set( parsed.keys() ) if miss_keys: return parsed, f"Missing keys: {miss_keys}" return parsed, None except JSONDecodeError as e: return {}, f"JSON decode error: {e}" Resource Management The Anthropic client is created for each call but not properly closed/cleaned up. Consider implementing proper resource management. client = anthropic.Anthropic(api_key=credentials.api_key.get_secret_value()) Input Validation The prompt_values handling assumes valid image data without thorough validation of the base64 content and MIME type. for key, value in values.items(): if isinstance(value, dict) and "content_type" in value and "data" in value: # This is an image content.append( { "type": "image", "source": { "type": "base64", "media_type": value["content_type"], "data": value["data"], }, } )

[Bentlybro]

i tried this and this is the full error

2025-01-24 14:21:34 Traceback (most recent call last):
2025-01-24 14:21:34   File "<frozen runpy>", line 198, in _run_module_as_main
2025-01-24 14:21:34   File "<frozen runpy>", line 88, in _run_code
2025-01-24 14:21:34   File "/app/autogpt_platform/backend/backend/rest.py", line 2, in <module>
2025-01-24 14:21:34     from backend.executor import DatabaseManager, ExecutionScheduler
2025-01-24 14:21:34   File "/app/autogpt_platform/backend/backend/executor/__init__.py", line 1, in <module>
2025-01-24 14:21:34     from .database import DatabaseManager
2025-01-24 14:21:34   File "/app/autogpt_platform/backend/backend/executor/database.py", line 4, in <module>
2025-01-24 14:21:34     from backend.data.credit import get_user_credit_model
2025-01-24 14:21:34   File "/app/autogpt_platform/backend/backend/data/credit.py", line 13, in <module>
2025-01-24 14:21:34     from backend.data.block_cost_config import BLOCK_COSTS
2025-01-24 14:21:34   File "/app/autogpt_platform/backend/backend/data/block_cost_config.py", line 3, in <module>
2025-01-24 14:21:34     from backend.blocks.ai_music_generator import AIMusicGeneratorBlock
2025-01-24 14:21:34   File "/app/autogpt_platform/backend/backend/blocks/__init__.py", line 24, in <module>
2025-01-24 14:21:34     importlib.import_module(f".{module}", package=__name__)
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
2025-01-24 14:21:34     return _bootstrap._gcd_import(name[level:], package, level)
2025-01-24 14:21:34            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/app/autogpt_platform/backend/backend/blocks/llm.py", line 1136, in <module>
2025-01-24 14:21:34     class ClaudeWithImageBlock(Block):
2025-01-24 14:21:34   File "/app/autogpt_platform/backend/backend/blocks/llm.py", line 1139, in ClaudeWithImageBlock
2025-01-24 14:21:34     class Input(BlockSchema):
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 226, in __new__
2025-01-24 14:21:34     complete_model_class(
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 658, in complete_model_class
2025-01-24 14:21:34     schema = cls.__get_pydantic_core_schema__(cls, handler)
2025-01-24 14:21:34              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/main.py", line 702, in __get_pydantic_core_schema__
2025-01-24 14:21:34     return handler(source)
2025-01-24 14:21:34            ^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py", line 84, in __call__
2025-01-24 14:21:34     schema = self._handler(source_type)
2025-01-24 14:21:34              ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 612, in generate_schema
2025-01-24 14:21:34     schema = self._generate_schema_inner(obj)
2025-01-24 14:21:34              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 881, in _generate_schema_inner
2025-01-24 14:21:34     return self._model_schema(obj)
2025-01-24 14:21:34            ^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 693, in _model_schema
2025-01-24 14:21:34     {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
2025-01-24 14:21:34     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 693, in <dictcomp>
2025-01-24 14:21:34     {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
2025-01-24 14:21:34         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1073, in _generate_md_field_schema
2025-01-24 14:21:34     common_field = self._common_field_schema(name, field_info, decorators)
2025-01-24 14:21:34                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1265, in _common_field_schema
2025-01-24 14:21:34     schema = self._apply_annotations(
2025-01-24 14:21:34              ^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 2062, in _apply_annotations
2025-01-24 14:21:34     schema = get_inner_schema(source_type)
2025-01-24 14:21:34              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py", line 84, in __call__
2025-01-24 14:21:34     schema = self._handler(source_type)
2025-01-24 14:21:34              ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 2043, in inner_handler
2025-01-24 14:21:34     schema = self._generate_schema_inner(obj)
2025-01-24 14:21:34              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 886, in _generate_schema_inner
2025-01-24 14:21:34     return self.match_type(obj)
2025-01-24 14:21:34            ^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 988, in match_type
2025-01-24 14:21:34     return self._match_generic_type(obj, origin)
2025-01-24 14:21:34            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1026, in _match_generic_type
2025-01-24 14:21:34     return self._dict_schema(*self._get_first_two_args_or_any(obj))
2025-01-24 14:21:34            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 372, in _dict_schema
2025-01-24 14:21:34     return core_schema.dict_schema(self.generate_schema(keys_type), self.generate_schema(values_type))
2025-01-24 14:21:34                                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 612, in generate_schema
2025-01-24 14:21:34     schema = self._generate_schema_inner(obj)
2025-01-24 14:21:34              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 886, in _generate_schema_inner
2025-01-24 14:21:34     return self.match_type(obj)
2025-01-24 14:21:34            ^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 988, in match_type
2025-01-24 14:21:34     return self._match_generic_type(obj, origin)
2025-01-24 14:21:34            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1016, in _match_generic_type
2025-01-24 14:21:34     return self._union_schema(obj)
2025-01-24 14:21:34            ^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1327, in _union_schema
2025-01-24 14:21:34     choices.append(self.generate_schema(arg))
2025-01-24 14:21:34                    ^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 612, in generate_schema
2025-01-24 14:21:34     schema = self._generate_schema_inner(obj)
2025-01-24 14:21:34              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 886, in _generate_schema_inner
2025-01-24 14:21:34     return self.match_type(obj)
2025-01-24 14:21:34            ^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 958, in match_type
2025-01-24 14:21:34     return self._typed_dict_schema(obj, None)
2025-01-24 14:21:34            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1408, in _typed_dict_schema
2025-01-24 14:21:34     raise PydanticUserError(
2025-01-24 14:21:34 pydantic.errors.PydanticUserError: Please use `typing_extensions.TypedDict` instead of `typing.TypedDict` on Python < 3.12.
2025-01-24 14:21:34 
2025-01-24 14:21:34 For further information visit https://errors.pydantic.dev/2.10/u/typed-dict-version

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[netlify[bot]]

✅ Deploy Preview for auto-gpt-docs-dev ready!

Name	Link
🔨 Latest commit	4c212f2b590ed27a8ef393d77f2774864a8f79fa
🔍 Latest deploy log	https://app.netlify.com/sites/auto-gpt-docs-dev/deploys/67f02a41ee3a220008ab7653
😎 Deploy Preview	https://deploy-preview-9309--auto-gpt-docs-dev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[netlify[bot]]

✅ Deploy Preview for auto-gpt-docs ready!

Name	Link
🔨 Latest commit	4c212f2b590ed27a8ef393d77f2774864a8f79fa
🔍 Latest deploy log	https://app.netlify.com/sites/auto-gpt-docs/deploys/67f02a412baed800089b3a0b
😎 Deploy Preview	https://deploy-preview-9309--auto-gpt-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[github-actions[bot]]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[deepsource-io[bot]]

Here's the code health analysis summary for commits 73d4331..4c212f2. View details on DeepSource ↗.

Analysis Summary

Analyzer	Status	Summary	Link
JavaScript	✅ Success		View Check ↗
Python	✅ Success	❗ 34 occurences introduced 🎯 17 occurences resolved	View Check ↗

---

💡 If you’re a repository administrator, you can configure the quality gates from the settings.

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[Pwuts]

I guess this is waiting to be integrated with #9320?

[ntindle]

Yeah time for that mostly

[github-actions[bot]]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[github-actions[bot]]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.