verl icon indicating copy to clipboard operation
verl copied to clipboard

[recipe] fix: langgraph_agent example robust math_expression.py for expression evaluation

Open erfgss opened this issue 1 week ago • 4 comments

Refactor calculate function to evaluate full expressions with custom operators and handle JSON input

What does this PR do?

Add concise overview of what this PR aims to achieve or accomplish. Reference related GitHub issues and PRs that help with the review.

This PR introduces a safe, deterministic math-expression evaluator with full support for:

  • Custom binary operator @ defined as 3a − 2b
  • Standard operators +, -, * with correct precedence
  • Parentheses & unary minus handling
  • Robust tokenization and RPN (Reverse Polish Notation) evaluation
  • Sanitization against malformed or partially-broken LLM tool outputs
  • Integration of this evaluator into a new MathExpressionReactAgentLoop
  • A LangChain-compatible tool calculate with safe JSON parsing fallback

This work replaces older, error-prone evaluation code and prevents agent crashes when LLM returns imperfect JSON in tool call arguments.

Related issues: (add links if available)


Checklist Before Starting

Example queries (already checked)

No existing PR found implementing safe math-evaluation + RPN + agent-loop tool integration.

Test

Since this PR focuses on improving the mathematical expression evaluation logic and stabilizing tool-call behavior in the ReactAgentLoop, its correctness cannot be validated through large-scale CI jobs alone. The following tests were conducted manually:

1. Functional correctness

Verified evaluation accuracy across:

  • Operator precedence (+, -, *, @)
  • Parentheses and nested expressions
  • Unary minus cases (-5, 3 + -2, (-3) * 4)
  • Custom operator semantics a @ b = 3a - 2b
  • Chained/mixed operator expressions

2. Robustness under malformed inputs

Evaluator handles:

  • Incomplete or noisy expressions
  • Mixed symbols / non-numeric characters
  • Empty / whitespace-only inputs
  • Incorrect or partially broken JSON
  • Dict-style tool inputs ({"expression": "..."})

Evaluator never raises exceptions and always returns a safe numeric value.

3. ReactAgentLoop integration

Tested the updated calculate tool:

  • Tool calls execute reliably without crashing
  • Invalid JSON from model is safely recovered
  • Output values match expected results

4. Formatting & static checks

Validated with:

  • pre-commit hooks (ruff, ruff-format, mypy)
  • No linting or typing issues
  • Formatting verified through CI

All tests passed as expected.


API and Usage Example

This PR introduces a safe mathematical expression evaluator and exposes it through calculate, automatically registered inside MathExpressionReactAgentLoop.

No public API changes.

Example

from recipe.langgraph_agent.example.math_expression import calculate

# Basic operators
calculate("3 + 5 * 2")      # 13
calculate("(3 + 5) * 2")    # 16

# Unary minus
calculate("-5 + 2")         # -3

# Custom operator '@' meaning: 3*a - 2*b
calculate("3 @ 2")          # 5

# Nested expressions
calculate("3 @ (9 @ 4 @ 4) @ (2 @ 2 @ 2)")

Design & Code Changes

This PR introduces a robust and safe mathematical expression evaluation pipeline tailored for LLM tool calls within the ReactAgentLoop.

Goals

  1. Deterministic & safe evaluation Avoid eval() — use:

    • Tokenization
    • Shunting-Yard algorithm
    • Reverse Polish Notation (RPN)
  2. Resilient to malformed LLM output Handles:

    • Broken JSON
    • Unexpected characters
    • Dict-style inputs
    • Partially invalid expressions
  3. Full support for custom operator @

    a @ b = 3*a − 2*b
    

    Precedence:

    • +, - → 1
    • *, @ → 2 All operators left-associative.
  4. Seamless ReactAgentLoop integration Exposed as a LangChain-style tool calculate.


Specific Code Changes

  • Added safe integer helper

    • Prevents crashes on invalid literal conversions.
  • Implemented full RPN evaluator (_eval_expr)

    • Cleans invalid characters
    • Supports unary minus
    • Nested parentheses
    • Shunting-Yard precedence handling
  • Enhanced tokenization

    • Normalizes whitespace
    • Recognizes unary minus
    • Removes invalid tokens safely
  • Added custom @ operator implementation

    • 3 * a - 2 * b
  • Updated calculate tool

    • Supports:

      • Raw strings
      • Dict-style arguments
      • Valid JSON
      • Recovery from malformed JSON

Checklist Before Submitting

IMPORTANT Please check all the following before requesting review.

  • [ ] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md).
  • [ ] Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
  • [ ] Add / update docs.
  • [ ] Add CI tests or explain why not feasible.
  • [ ] After ready for CI, notify ci-request channel in Slack/Feishu.

erfgss avatar Nov 17 '25 03:11 erfgss

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Nov 17 '25 03:11 CLAassistant

@wuxibin89 PTAL, thx.

david6666666 avatar Nov 18 '25 11:11 david6666666

@erfgss Please format code: https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting

wuxibin89 avatar Nov 19 '25 04:11 wuxibin89

@erfgss Please format code: https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting

It has been modified. Please take a look.

erfgss avatar Nov 21 '25 08:11 erfgss