Real-world prompt library for the `ov.Agent`

[HendricksJudy]

This pull request introduces several improvements to the Python package workflow and documentation for the OvIntelligence project. The most significant changes include updating the installation steps to simplify dependencies, enhancing the GitHub Actions workflow for Python packaging, and adding a comprehensive real-world prompt library for the ov.Agent to facilitate workflow reproduction.

Dependency and installation improvements:

• The README (OvIntelligence/README.md) has been updated to remove the Pantheon agent framework installation step, reflecting a switch to an internal backend for smart agent integration. This simplifies setup and avoids unnecessary dependencies for users.

• The GitHub Actions workflow (.github/workflows/python-package.yml) now installs pytest-asyncio and ensures the local package is installed in editable mode, overriding any PyPI version specified in requirements.txt. This improves testing flexibility and ensures the latest local changes are always used in CI.

Documentation and usability enhancements:

• A new prompt library file (OvIntelligence/ov_agent_realworld_prompts.md) has been added. This file provides scenario-driven prompts for ov.Agent and maps them to each tutorial notebook in omicverse_guide/docs. These prompts help users reproduce or adapt workflows for real-world biological data analysis without needing to reference notebook filenames, improving usability and onboarding.

[HendricksJudy]

@claude

[HendricksJudy]

Happy to Pull a Request. Go to sleep now

[claude[bot]]

Claude finished @HendricksJudy's task —— View job

---

PR Review: Real-world prompt library for ov.Agent

This is a substantial PR with 100 changed files and 39k+ additions introducing major enhancements to the OmicVerse Agent system. Below is my comprehensive review.

---

Summary

This PR introduces:

1. Simplified installation - Removes external Pantheon dependency in favor of internal backend
2. Enhanced CI workflow - Better test coverage with pytest-asyncio
3. New prompt library - Real-world scenario prompts for ov.Agent
4. Major architectural expansion - New inspector/verifier modules, expanded agent backend

---

⚠️ Critical Issues

1. CI Failures (MUST FIX)

• ✅ Python 3.11: Passing
• ❌ Python 3.10: Failing in pytest step
• Action Required: The Python 3.10 test failures must be resolved before merge. This could indicate compatibility issues that will affect users.

2. Dependency Confusion in Documentation

The PR description states:

"removing the Pantheon agent framework installation step, reflecting a switch to an internal backend"

However, in OvIntelligence/README.md (lines 17-24), the instructions still require Pantheon:

pip install pantheon-agents

And in omicverse/utils/smart_agent.py:99-100:

print(f"❌ Pantheon not found. Please install pantheon-agents...")
raise ImportError("Pantheon not found...")

Recommendation: Either:

• Update the README to reflect that Pantheon is still required, OR
• Complete the migration to the internal backend and remove Pantheon dependency entirely

This inconsistency will confuse users.

3. Missing Files in Repository

According to <changed_files>, these new files should exist but don't:

• OvIntelligence/ov_agent_realworld_prompts.md ⭐ (main deliverable!)
• OvIntelligence/ov_agent_tutorial_data_sources.md
• omicverse/utils/inspector/*.py (entire module)
• omicverse/utils/verifier/*.py (entire module)
• omicverse/utils/agent_backend.py
• Multiple test files in tests/utils/inspector/

This is concerning - the core feature (real-world prompt library) is listed as added but doesn't exist in the branch. This suggests:

• The PR branch might not be properly pushed
• These files exist in a different branch
• There's a synchronization issue

---

Code Quality Observations

Positive Aspects ✅

1. Well-structured registry system (omicverse/utils/registry.py):

   • Clean dataclass-based design
   • Good validation in register() method (lines 62-81)
   • Comprehensive metadata tracking

2. LLM provider abstraction (omicverse/utils/skill_registry.py):

   • Smart provider-specific formatting (lines 21-41)
   • Good separation of concerns with SkillInstructionFormatter

3. Async/threading handling (smart_agent.py:840-865):

   • Proper detection of existing event loops
   • Thread-based workaround for nested async calls

Areas of Concern ⚠️

1. Error handling in smart_agent.py:

   except ImportError:
       PANTHEON_INSTALLED = False
   

   This silently catches ALL import errors, not just Pantheon. Consider:

   except ModuleNotFoundError:
       PANTHEON_INSTALLED = False
   

2. Incomplete migration warning: The codebase still heavily references Pantheon (line 277: "Setup the pantheon agent...") despite claiming to move to an internal backend.

3. Missing prompt library validation: If ov_agent_realworld_prompts.md is the main deliverable, there should be:

   • Validation tests to ensure prompts are well-formed
   • Documentation on prompt structure
   • Examples of expected outcomes

---

Architectural Concerns

Scale of Changes

This PR modifies 100 files with significant additions to core modules:

• smart_agent.py: Expanded significantly
• New inspector/ module: 13+ files
• New verifier/ module: 10+ files
• New test suites: 20+ files

Recommendation: Consider breaking this into smaller PRs:

1. PR 1: Infrastructure changes (registry updates, model config)
2. PR 2: Inspector/verifier modules with tests
3. PR 3: Prompt library and documentation
4. PR 4: CI/workflow updates

This would make review more manageable and reduce risk of introducing bugs.

Missing Documentation

For a feature of this scale, I'd expect:

• Architecture documentation (how inspector/verifier work)
• Migration guide (if moving away from Pantheon)
• API documentation for new modules
• Tutorial updates showing the new prompt library usage

---

GitHub Actions Workflow Review

From .github/workflows/python-package.yml, the changes mentioned in the PR description should add:

• pytest-asyncio installation
• Editable mode installation

However, the current file (lines 28-31) doesn't show these changes:

python -m pip install flake8 pytest torch tensorboard scrublet mellon torch_geometric POT wget
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi

Missing:

• No pytest-asyncio in the install step
• No -e . for editable mode installation

This contradicts the PR description.

---

Security Considerations

Code Execution Risk

OvIntelligence/README.md:75-78 mentions:

"The OmicVerse smart agent executes generated Python inside a restricted sandbox"

Questions:

1. What sandboxing mechanism is used?
2. Is it properly tested against injection attacks?
3. Are there audit logs for code execution?

Given that LLM-generated code will be executed, this needs thorough security review.

---

Testing Gaps

Based on the test file list, I see good coverage for:

• ✅ Agent initialization
• ✅ Skill instruction formatting
• ✅ Model normalization

Missing tests for:

• ❌ Real-world prompt library (the main feature!)
• ❌ Inspector module validation
• ❌ Verifier end-to-end workflows
• ❌ Security/sandboxing mechanisms
• ❌ Error handling edge cases

---

Recommendations

Must Do Before Merge

1. ✅ Fix Python 3.10 CI failures
2. ✅ Resolve Pantheon dependency confusion - clarify if it's required or not
3. ✅ Verify all listed files exist in the branch - especially the prompt library
4. ✅ Update GitHub Actions to match PR description (pytest-asyncio, editable mode)

Should Do

5. Add tests for the prompt library functionality
6. Document the inspector/verifier architecture
7. Security review of code execution sandbox
8. Break into smaller PRs for easier review

Nice to Have

9. Add examples of real-world prompt usage in tutorials
10. Create migration guide if moving from Pantheon
11. Add validation for prompt structure/format

---

Questions for Author

1. Where is ov_agent_realworld_prompts.md? This is listed as the main deliverable but doesn't appear to exist.
2. Pantheon status? Is it still required or has it been replaced? Documentation conflicts.
3. Why Python 3.10 failures? What changed to break Python 3.10 compatibility?
4. Inspector/verifier modules? What's their purpose and why are they needed?
5. Breaking changes? Are there any API changes users need to know about?

---

Verdict

Status: ⚠️ Needs Work - Cannot approve due to:

• CI failures
• Missing core files (prompt library)
• Documentation inconsistencies
• Unclear dependency status

This is ambitious and valuable work, but needs refinement before merge. The prompt library concept is excellent and will greatly improve usability, but the implementation needs to be complete and tested.

---

Effort Estimate: 5-10 hours to address critical issues Risk Level: High (large-scale changes, missing files, CI failures)

[Starlitnightly]

@HendricksJudy

The failure is due to AttributeError: module 'omicverse.utils' has no attribute 'verifier' in tests that patch or import omicverse.utils.verifier. This happens because, starting in Python 3.10, submodules do not become attributes of the parent module by default, so omicverse.utils.verifier is not directly accessible unless explicitly set.

Solution: Your omicverse/utils/init.py already has code to handle this, but ensure the following lines are present and correct:

from . import verifier as _verifier_module

def __getattr__(name):
    if name == 'verifier':
        return _verifier_module
    raise AttributeError(f"module '{__name__}' has no attribute '{name}'")

verifier = _verifier_module

This setup makes omicverse.utils.verifier accessible both for imports and for patching in tests. If you still get this error, double-check for typos or import errors in omicverse/utils/init.py, and make sure tests and any code importing verifier use one of the following patterns:

• Import submodule directly:

  from omicverse.utils import verifier
  

• Or access via the utils namespace:

  import omicverse.utils
  omicverse.utils.verifier
  

Once this import shim is active, your tests that patch omicverse.utils.verifier should pass.

Reference: See the code in omicverse/utils/init.py at ref 22b877c9e89cc4e6da0ff82c02b8e2daa8da3ae0 for the import logic and patching support.

[Starlitnightly]

@HendricksJudy I have another question, do you calculate the Agent response's speed?

[HendricksJudy]

@HendricksJudy

The failure is due to AttributeError: module 'omicverse.utils' has no attribute 'verifier' in tests that patch or import omicverse.utils.verifier. This happens because, starting in Python 3.10, submodules do not become attributes of the parent module by default, so omicverse.utils.verifier is not directly accessible unless explicitly set.

Solution: Your omicverse/utils/init.py already has code to handle this, but ensure the following lines are present and correct:

from . import verifier as _verifier_module

def __getattr__(name):
    if name == 'verifier':
        return _verifier_module
    raise AttributeError(f"module '{__name__}' has no attribute '{name}'")

verifier = _verifier_module

This setup makes omicverse.utils.verifier accessible both for imports and for patching in tests. If you still get this error, double-check for typos or import errors in omicverse/utils/init.py, and make sure tests and any code importing verifier use one of the following patterns:

• Import submodule directly:

  from omicverse.utils import verifier
  

• Or access via the utils namespace:

  import omicverse.utils
  omicverse.utils.verifier
  

Once this import shim is active, your tests that patch omicverse.utils.verifier should pass.

Reference: See the code in omicverse/utils/init.py at ref 22b877c for the import logic and patching support.

still failed. I am very confused

[HendricksJudy]

@HendricksJudy I have another question, do you calculate the Agent response's speed?

I wil, this module I try to add it when the agent become stale

[HendricksJudy]

I am going to let ov.agent become notebook-aware

[HendricksJudy]

I am going to let ov.agent become notebook-aware

This target has been achieved for now ov.agent is stable and useful