feat: Merge mgx_ops, framework part.

[iorisa]

Features

1. Fixbug: When the context of Environment object changes, the Context object of the Action object of Role object does not change synchronously;
2. Refactor: CodeParser.parse_code
3. Refactor: Replace Message and ActionOutput with UserMessage, AIMessage, SystemMessage;
4. Feat: Replace input with get_human_input to support custom input api;
5. Refactor: Refactor the modification of git_repo and repo in Context;
6. Feat: Replace the implementation method async def _observe(self, ignore_memory=False) with Role.enable_memory. Stateless, atomic roles, or roles that use external storage can disable this to save memory.
7. Feat: Add get_project_srcs_path to simplify the codes to find the source code directory of the software project.
8. Feat: Role.publish_message adds the ability to auto-fill the default send_to and sent_from values.
9. Refactor: Environment.run;
10. Feat: Add Message.metadata for content and instruct_content metadata.
11. Feat: Add Message.parse_resources. parse_resources corresponds to the in-context adaptation capability of the input of the atomic action, which will be migrated to the context builder later.
12. Feat: Add tools to read heterogeneous data.
13. Fixbug: unit tests

[qodo-code-review[bot]]

PR-Agent was enabled for this repository. To continue using it, please link your git user with your CodiumAI identity here.

PR Description updated to latest commit (https://github.com/geekan/MetaGPT/commit/5f8b7e85c0a10636a7b0488878bd90c03881ab71)

• [ ] Copy walkthrough table to "Files Changed" Tab

[qodo-code-review[bot]]

PR-Agent was enabled for this repository. To continue using it, please link your git user with your CodiumAI identity here.

PR Review 🔍

⏱️ Estimated effort to review [1-5]	4, because the PR involves a significant number of files and changes across various modules, including action classes, utility functions, and schema updates. The changes include refactoring, bug fixes, and enhancements which require a thorough review to ensure compatibility and functionality.
🧪 Relevant tests	No
⚡ Possible issues	Possible Bug: The method override_context in action.py sets private_context to context only if private_context is not already set. This might not correctly handle cases where the context needs to be updated after initialization.
	Refactoring Concern: The removal of parameters like block in CodeParser.parse_code across multiple files might affect other parts of the system that rely on this parameter.
🔒 Security concerns	No

[qodo-code-review[bot]]

PR-Agent was enabled for this repository. To continue using it, please link your git user with your CodiumAI identity here.

PR Code Suggestions ✨

Category	Suggestion	Score
Possible issue	Prevent creating branches with invalid names by checking for empty or whitespace-only names Consider handling the case where the branch_name is empty or contains only whitespace. This can prevent creating branches with invalid names. metagpt/utils/git_repository.py [229-230] -if not branch_name: - return self.current_branch +if not branch_name.strip(): + raise ValueError("Branch name cannot be empty or whitespace") Suggestion importance[1-10]: 8 Why: The suggestion correctly identifies a potential issue where branches with invalid names could be created. Implementing this check enhances the robustness of branch creation.	8
	Validate the new_branch parameter to ensure it is not empty before use Validate the new_branch parameter to ensure it is not empty before attempting to push changes, which can prevent runtime errors and unintended behavior. metagpt/utils/git_repository.py [274-275] +if not new_branch.strip(): + raise ValueError("New branch name cannot be empty.") base = self.current_branch head = base if not new_branch else self.new_branch(new_branch) Suggestion importance[1-10]: 8 Why: This suggestion prevents potential runtime errors or unintended behavior by ensuring that the new_branch parameter is not empty before proceeding with branch operations, which is crucial for maintaining code integrity.	8
	Ensure that messages created with AIMessage contain meaningful content or reconsider their usage The AIMessage class is used to create messages, but the content field is often set to an empty string, which might indicate that the messages are not being utilized effectively. Consider either populating the content field with meaningful data or, if the messages are indeed not needed, reconsider the design to potentially remove unnecessary message creations. metagpt/roles/engineer.py [221] -return AIMessage(content="", cause_by=WriteCodePlanAndChange, send_to=MESSAGE_ROUTE_TO_SELF) +return AIMessage(content="Code plan and change completed.", cause_by=WriteCodePlanAndChange, send_to=MESSAGE_ROUTE_TO_SELF) Suggestion importance[1-10]: 5 Why: The suggestion is valid as it encourages meaningful use of the content field in AIMessage. However, the impact of this change is moderate since it primarily affects readability and clarity rather than functionality or performance.	5
Possible bug	Validate the type of json_file before usage to ensure it's either a string or a Path object Consider checking if the json_file parameter is a valid path or string before attempting to open it. This can prevent potential errors when the json_file is neither a string nor a Path object. metagpt/utils/common.py [564] -if not json_file or not Path(json_file).exists(): +if not isinstance(json_file, (str, Path)) or not Path(json_file).exists(): Suggestion importance[1-10]: 8 Why: This suggestion correctly identifies a potential bug where the json_file parameter might not be a valid type, which could lead to runtime errors. The suggested improvement robustly checks the type before proceeding, which is crucial for preventing errors in file handling operations.	8
	Ensure msg.send_to is a mutable set before modification in publish_message In the publish_message method, ensure that msg.send_to is a mutable set before attempting to modify it, to avoid potential errors if send_to is not set or is immutable. metagpt/roles/role.py [448] -if MESSAGE_ROUTE_TO_SELF in msg.send_to: +if hasattr(msg, 'send_to') and isinstance(msg.send_to, set) and MESSAGE_ROUTE_TO_SELF in msg.send_to: Suggestion importance[1-10]: 8 Why: The suggestion addresses a potential bug where msg.send_to might not be initialized or mutable, which could lead to runtime errors when attempting to modify it. Ensuring it's a mutable set before modification is crucial for the stability of the message handling logic.	8
	Add a null check before iterating over a dictionary to prevent potential runtime errors Consider adding a check for None before iterating over key_descriptions in the parse_resources method to prevent potential TypeError if None is passed explicitly. metagpt/schema.py [349] -for k, v in key_descriptions.items(): +if key_descriptions is not None: + for k, v in key_descriptions.items(): Suggestion importance[1-10]: 8 Why: This suggestion correctly identifies a potential bug where a None value could cause a TypeError during iteration, and the fix prevents this error, enhancing the method's robustness.	8
	Handle missing remote origin to prevent AttributeError Add error handling for the case where self._repository.remotes.origin might not exist, which would raise an AttributeError when trying to access .url. metagpt/utils/git_repository.py [202] -return self._repository.remotes.origin.url +try: + return self._repository.remotes.origin.url +except AttributeError: + logger.error("Remote origin not set.") + return "" Suggestion importance[1-10]: 7 Why: This suggestion addresses a common error scenario where the remote origin might not be set, which would otherwise lead to an AttributeError. Adding error handling improves the code's reliability.	7
	Add error handling for potentially unset or invalid work directories when setting src_workspace The method _think sets self.src_workspace using get_project_srcs_path, which is a good use of utility functions. However, ensure that get_project_srcs_path handles cases where workdir might not be set or is invalid, to prevent runtime errors. metagpt/roles/engineer.py [232] -self.src_workspace = get_project_srcs_path(self.project_repo.workdir) +if self.project_repo.workdir: + self.src_workspace = get_project_srcs_path(self.project_repo.workdir) +else: + logger.error("Project work directory is not set.") Suggestion importance[1-10]: 7 Why: This suggestion addresses a potential bug by adding error handling for unset or invalid work directories. It's crucial for preventing runtime errors, thus improving the robustness of the code.	7
Enhancement	Add error handling for JSON parsing to improve the robustness of the method The parse_resources method currently does not handle exceptions that may occur during JSON parsing. It is recommended to add error handling around the json.loads call to manage potential parsing errors gracefully. metagpt/schema.py [354-355] json_data = CodeParser.parse_code(text=rsp, lang="json") -m = json.loads(json_data) +try: + m = json.loads(json_data) +except json.JSONDecodeError as e: + logger.error(f"Failed to parse JSON data: {e}") + return {} Suggestion importance[1-10]: 8 Why: Adding error handling for JSON parsing is crucial for robustness, especially in a method that deals with external data, making this suggestion highly relevant and impactful.	8
	Improve the extraction of repository name from URL using regex for better error handling Use a more robust method to extract the repository name from the URL to handle edge cases and potential errors more gracefully. metagpt/utils/git_repository.py [212-215] -if self.remote_url.startswith("https://"): - return self.remote_url.split("/", maxsplit=3)[-1].replace(".git", "") -elif self.remote_url.startswith("git@"): - return self.remote_url.split(":")[-1].replace(".git", "") +import re +match = re.search(r'\/([^\/]+)\.git$', self.remote_url) +if match: + return match.group(1) +else: + logger.error("Invalid remote URL format.") + return "" Suggestion importance[1-10]: 7 Why: Using regex for parsing the repository name from the URL is more robust and handles more edge cases than simple string operations. This suggestion enhances the method's accuracy and error handling.	7
	Validate dictionary keys before initializing class instances to ensure data integrity To ensure that the Resource class instances are correctly initialized with all required fields, consider validating the presence of all keys ('resource_type', 'value', 'description') in the dictionary before creating Resource objects. metagpt/schema.py [356] -m["resources"] = [Resource(**i) for i in m.get("resources", [])] +m["resources"] = [Resource(**i) for i in m.get("resources", []) if 'resource_type' in i and 'value' in i and 'description' in i] Suggestion importance[1-10]: 7 Why: This suggestion improves data integrity by ensuring that all required keys are present before object instantiation, which is a good practice to avoid runtime errors.	7
	Add specific exception handling for regex operations in parse_code To improve the robustness of the parse_code method, consider adding a more specific exception handling around the regex pattern usage to catch and handle potential regex-related errors. metagpt/utils/common.py [277] -pattern = rf"```{lang}.*?\s+(.*?)```" +try: + pattern = rf"```{lang}.*?\s+(.*?)```" +except re.error as e: + logger.error(f"Regex error in parse_code with lang={lang}: {str(e)}") + return "" Suggestion importance[1-10]: 6 Why: The suggestion to add specific exception handling for regex operations in the parse_code method is valid and enhances the robustness of the method. However, it's a moderate improvement as the primary functionality likely remains correct without it.	6
	Prevent accidental overwriting of metadata by checking for existing keys before insertion Replace the direct manipulation of the metadata dictionary in the add_metadata method with a method that checks for existing keys to prevent accidental overwriting of metadata. metagpt/schema.py [360] -self.metadata[key] = value +if key not in self.metadata: + self.metadata[key] = value +else: + logger.warning(f"Key {key} already exists in metadata.") Suggestion importance[1-10]: 6 Why: The suggestion to prevent accidental overwriting by checking for existing keys is valid and improves the method's safety, although it's a relatively minor enhancement.	6
Best practice	Use the constant MESSAGE_ROUTE_TO_SELF consistently across all message sending functions The MESSAGE_ROUTE_TO_SELF constant is used in multiple places to specify the recipient of messages. To maintain consistency and avoid hardcoding the recipient in multiple places, consider using this constant in all instances where the message is intended to be routed to self. metagpt/roles/engineer.py [157] +send_to=MESSAGE_ROUTE_TO_SELF - Suggestion importance[1-10]: 8 Why: The suggestion correctly identifies the use of MESSAGE_ROUTE_TO_SELF to maintain consistency and avoid hardcoding, which is a best practice in coding to reduce errors and improve maintainability.	8
Maintainability	Refactor string to Path conversion into a separate helper function in encode_image Refactor the encode_image function to separate concerns by creating a helper function for handling the conversion of image paths to Path objects. This improves code readability and maintainability. metagpt/utils/common.py [796-797] -if isinstance(image_path_or_pil, str): - image_path_or_pil = Path(image_path_or_pil) +def ensure_path(image_input): + return Path(image_input) if isinstance(image_input, str) else image_input +image_path_or_pil = ensure_path(image_path_or_pil) + Suggestion importance[1-10]: 7 Why: This suggestion correctly identifies an opportunity to improve code maintainability by refactoring repeated logic into a helper function. This change would make the encode_image function cleaner and more maintainable.	7
	Improve readability and maintainability of string constructions in message content The method _act_summarize constructs a message with a concatenated string that includes paths and filenames. This construction could be more readable and less error-prone by using f-string formatting throughout, especially in complex string constructions. metagpt/roles/engineer.py [187-192] +return AIMessage( + content=f"Coding is complete. The source code is at {self.project_repo.workdir.name}/{self.project_repo.srcs.root_path}, containing: " + + "\n".join( + list(self.project_repo.resources.code_summary.changed_files.keys()) + + list(self.project_repo.srcs.changed_files.keys()) + ), + cause_by=SummarizeCode, + send_to="Edward", +) - Suggestion importance[1-10]: 6 Why: The suggestion to use f-string for complex string constructions improves readability and maintainability, which is beneficial for future code modifications and debugging.	6

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 50.80214% with 276 lines in your changes are missing coverage. Please review.

Project coverage is 68.39%. Comparing base (7057ace) to head (6b98fb8). Report is 100 commits behind head on main.

Files	Patch %	Lines
metagpt/utils/git_repository.py	29.19%	114 Missing :warning:
metagpt/tools/libs/env.py	0.00%	79 Missing :warning:
metagpt/schema.py	54.38%	26 Missing :warning:
metagpt/utils/common.py	74.50%	13 Missing :warning:
metagpt/utils/project_repo.py	42.85%	8 Missing :warning:
metagpt/roles/role.py	83.33%	5 Missing :warning:
metagpt/utils/repo_to_markdown.py	89.36%	5 Missing :warning:
metagpt/roles/engineer.py	50.00%	4 Missing :warning:
metagpt/utils/file_repository.py	0.00%	4 Missing :warning:
metagpt/utils/s3.py	33.33%	4 Missing :warning:
... and 8 more

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1273      +/-   ##
==========================================
- Coverage   70.20%   68.39%   -1.82%     
==========================================
  Files         316      319       +3     
  Lines       18860    19263     +403     
==========================================
- Hits        13240    13174      -66     
- Misses       5620     6089     +469     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.