llm-math Tools takes a word problem rather than a numerical expression.

[Hizafa-Nadeem]

System Info

Working on Windows machine using a conda environment.

aiohttp==3.8.4 aiosignal==1.3.1 altair==4.2.2 anyio @ file:///C:/ci_311/anyio_1676425491996/work/dist argilla==1.6.0 argon2-cffi @ file:///opt/conda/conda-bld/argon2-cffi_1645000214183/work argon2-cffi-bindings @ file:///C:/ci_311/argon2-cffi-bindings_1676424443321/work asttokens @ file:///opt/conda/conda-bld/asttokens_1646925590279/work async-timeout==4.0.2 attrs @ file:///C:/ci_311/attrs_1676422272484/work Babel @ file:///C:/ci_311/babel_1676427169844/work backcall @ file:///home/ktietz/src/ci/backcall_1611930011877/work backoff==2.2.1 beautifulsoup4 @ file:///C:/b/abs_0agyz1wsr4/croot/beautifulsoup4-split_1681493048687/work bleach @ file:///opt/conda/conda-bld/bleach_1641577558959/work blinker==1.6.2 brotlipy==0.7.0 cachetools==5.3.0 certifi @ file:///C:/b/abs_4a0polqwty/croot/certifi_1683875377622/work/certifi cffi @ file:///C:/ci_311/cffi_1676423759166/work chardet==5.1.0 charset-normalizer @ file:///tmp/build/80754af9/charset-normalizer_1630003229654/work chromadb==0.3.21 click==8.1.3 clickhouse-connect==0.5.22 colorama @ file:///C:/ci_311/colorama_1676422310965/work comm @ file:///C:/ci_311/comm_1678376562840/work commonmark==0.9.1 cryptography @ file:///C:/ci_311/cryptography_1679419210767/work Cython==0.29.34 dataclasses-json==0.5.7 debugpy @ file:///C:/ci_311/debugpy_1676426137692/work decorator @ file:///opt/conda/conda-bld/decorator_1643638310831/work defusedxml @ file:///tmp/build/80754af9/defusedxml_1615228127516/work Deprecated==1.2.13 duckdb==0.7.1 entrypoints @ file:///C:/ci_311/entrypoints_1676423328987/work et-xmlfile==1.1.0 executing @ file:///opt/conda/conda-bld/executing_1646925071911/work fastapi==0.95.1 fastjsonschema @ file:///C:/ci_311/python-fastjsonschema_1679500568724/work filelock==3.12.0 frozenlist==1.3.3 fsspec==2023.4.0 ftfy==6.1.1 gitdb==4.0.10 GitPython==3.1.31 google-api-core==2.11.0 google-api-python-client==2.86.0 google-auth==2.18.0 google-auth-httplib2==0.1.0 googleapis-common-protos==1.59.0 greenlet==2.0.2 h11==0.14.0 hnswlib @ file:///D:/bld/hnswlib_1675802891722/work httpcore==0.16.3 httplib2==0.22.0 httptools==0.5.0 httpx==0.23.3 huggingface-hub==0.14.1 idna @ file:///C:/ci_311/idna_1676424932545/work importlib-metadata==6.6.0 ipykernel @ file:///C:/ci_311/ipykernel_1678734799670/work ipython @ file:///C:/b/abs_d1yx5tjhli/croot/ipython_1680701887259/work ipython-genutils @ file:///tmp/build/80754af9/ipython_genutils_1606773439826/work ipywidgets @ file:///C:/b/abs_5awapknmz_/croot/ipywidgets_1679394824767/work jedi @ file:///C:/ci_311/jedi_1679427407646/work Jinja2 @ file:///C:/ci_311/jinja2_1676424968965/work joblib==1.2.0 json5 @ file:///tmp/build/80754af9/json5_1624432770122/work jsonschema @ file:///C:/b/abs_d40z05b6r1/croot/jsonschema_1678983446576/work jupyter @ file:///C:/ci_311/jupyter_1678249952587/work jupyter-console @ file:///C:/b/abs_82xaa6i2y4/croot/jupyter_console_1680000189372/work jupyter-server @ file:///C:/ci_311/jupyter_server_1678228762759/work jupyter_client @ file:///C:/b/abs_059idvdagk/croot/jupyter_client_1680171872444/work jupyter_core @ file:///C:/b/abs_9d0ttho3bs/croot/jupyter_core_1679906581955/work jupyterlab @ file:///C:/ci_311/jupyterlab_1677719392688/work jupyterlab-pygments @ file:///tmp/build/80754af9/jupyterlab_pygments_1601490720602/work jupyterlab-widgets @ file:///C:/b/abs_38ad427jkz/croot/jupyterlab_widgets_1679055289211/work jupyterlab_server @ file:///C:/b/abs_e0qqsihjvl/croot/jupyterlab_server_1680792526136/work langchain==0.0.157 lxml @ file:///C:/b/abs_c2bg6ck92l/croot/lxml_1679646459966/work lz4==4.3.2 Markdown==3.4.3 MarkupSafe @ file:///C:/ci_311/markupsafe_1676424152318/work marshmallow==3.19.0 marshmallow-enum==1.5.1 matplotlib-inline @ file:///C:/ci_311/matplotlib-inline_1676425798036/work mistune @ file:///C:/ci_311/mistune_1676425149302/work monotonic==1.6 mpmath==1.3.0 msg-parser==1.2.0 multidict==6.0.4 mypy-extensions==1.0.0 nbclassic @ file:///C:/b/abs_c8_rs7b3zw/croot/nbclassic_1681756186106/work nbclient @ file:///C:/ci_311/nbclient_1676425195918/work nbconvert @ file:///C:/ci_311/nbconvert_1676425836196/work nbformat @ file:///C:/ci_311/nbformat_1676424215945/work nest-asyncio @ file:///C:/ci_311/nest-asyncio_1676423519896/work networkx==3.1 nltk==3.8.1 notebook @ file:///C:/b/abs_49d8mc_lpe/croot/notebook_1681756182078/work notebook_shim @ file:///C:/ci_311/notebook-shim_1678144850856/work numexpr==2.8.4 numpy==1.23.5 olefile==0.46 openai==0.27.6 openapi-schema-pydantic==1.2.4 openpyxl==3.1.2 packaging @ file:///C:/b/abs_ed_kb9w6g4/croot/packaging_1678965418855/work pandas==1.5.3 pandocfilters @ file:///opt/conda/conda-bld/pandocfilters_1643405455980/work parso @ file:///opt/conda/conda-bld/parso_1641458642106/work pdfminer.six==20221105 pickleshare @ file:///tmp/build/80754af9/pickleshare_1606932040724/work Pillow==9.5.0 platformdirs @ file:///C:/ci_311/platformdirs_1676422658103/work ply==3.11 posthog==3.0.1 prometheus-client @ file:///C:/ci_311/prometheus_client_1679591942558/work prompt-toolkit @ file:///C:/ci_311/prompt-toolkit_1676425940920/work protobuf==3.20.3 psutil @ file:///C:/ci_311_rebuilds/psutil_1679005906571/work pure-eval @ file:///opt/conda/conda-bld/pure_eval_1646925070566/work pyarrow==11.0.0 pyasn1==0.5.0 pyasn1-modules==0.3.0 pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work pydantic==1.10.7 pydeck==0.8.1b0 Pygments @ file:///opt/conda/conda-bld/pygments_1644249106324/work Pympler==1.0.1 pyOpenSSL @ file:///C:/b/abs_de215ipd18/croot/pyopenssl_1678965319166/work pypandoc==1.11 pyparsing==3.0.9 PyQt5==5.15.7 PyQt5-sip @ file:///C:/ci_311/pyqt-split_1676428895938/work/pyqt_sip pyrsistent @ file:///C:/ci_311/pyrsistent_1676422695500/work PySocks @ file:///C:/ci_311/pysocks_1676425991111/work python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work python-docx==0.8.11 python-dotenv==1.0.0 python-magic==0.4.27 python-pptx==0.6.21 pytz @ file:///C:/ci_311/pytz_1676427070848/work pytz-deprecation-shim==0.1.0.post0 pywin32==305.1 pywinpty @ file:///C:/ci_311/pywinpty_1677707791185/work/target/wheels/pywinpty-2.0.10-cp311-none-win_amd64.whl PyYAML==6.0 pyzmq @ file:///C:/b/abs_8b16zbmf46/croot/pyzmq_1682697651374/work qtconsole @ file:///C:/b/abs_eb4u9jg07y/croot/qtconsole_1681402843494/work QtPy @ file:///C:/ci_311/qtpy_1676432558504/work regex==2023.3.23 requests @ file:///C:/b/abs_41owkd5ymz/croot/requests_1682607524657/work rfc3986==1.5.0 rich==13.0.1 rsa==4.9 scikit-learn==1.2.2 scipy==1.10.1 Send2Trash @ file:///tmp/build/80754af9/send2trash_1632406701022/work sentence-transformers==2.2.2 sentencepiece==0.1.99 sip @ file:///C:/ci_311/sip_1676427825172/work six @ file:///tmp/build/80754af9/six_1644875935023/work smmap==5.0.0 sniffio @ file:///C:/ci_311/sniffio_1676425339093/work soupsieve @ file:///C:/b/abs_a989exj3q6/croot/soupsieve_1680518492466/work SQLAlchemy==2.0.12 stack-data @ file:///opt/conda/conda-bld/stack_data_1646927590127/work starlette==0.26.1 streamlit==1.22.0 sympy==1.11.1 tenacity==8.2.2 terminado @ file:///C:/ci_311/terminado_1678228513830/work threadpoolctl==3.1.0 tiktoken==0.3.3 tinycss2 @ file:///C:/ci_311/tinycss2_1676425376744/work tokenizers==0.13.3 toml @ file:///tmp/build/80754af9/toml_1616166611790/work tomli @ file:///C:/ci_311/tomli_1676422027483/work toolz==0.12.0 torch==2.0.0 torchvision==0.15.1 tornado @ file:///C:/ci_311/tornado_1676423689414/work tqdm==4.65.0 traitlets @ file:///C:/ci_311/traitlets_1676423290727/work transformers==4.28.1 typing-inspect==0.8.0 typing_extensions @ file:///C:/b/abs_a1bb332wcs/croot/typing_extensions_1681939523095/work tzdata==2023.3 tzlocal==4.3 unstructured==0.6.2 uritemplate==4.1.1 urllib3 @ file:///C:/b/abs_3ce53vrdcr/croot/urllib3_1680254693505/work uvicorn==0.22.0 validators==0.20.0 watchdog==3.0.0 watchfiles==0.19.0 wcwidth @ file:///Users/ktietz/demo/mc3/conda-bld/wcwidth_1629357192024/work webencodings==0.5.1 websocket-client @ file:///C:/ci_311/websocket-client_1676426063281/work websockets==11.0.2 widgetsnbextension @ file:///C:/b/abs_882k4_4kdf/croot/widgetsnbextension_1679313880295/work wikipedia==1.4.0 win-inet-pton @ file:///C:/ci_311/win_inet_pton_1676425458225/work wrapt==1.14.1 XlsxWriter==3.1.0 yarl==1.9.2 zipp==3.15.0 zstandard==0.21.0

Who can help?

@vowelparrot

Hi, I facing an issue regarding llm-math tool. I am using a Tool that is created using my documents and a built-in llm tool. For some queries, llm tool receives a word problem rather than a mathematical expression. Kindly, guide how can I avoid this problem?

Information

• [ ] The official example notebooks/scripts
• [X] My own modified scripts

Related Components

• [X] LLMs/Chat Models
• [X] Embedding Models
• [X] Prompts / Prompt Templates / Prompt Selectors
• [ ] Output Parsers
• [ ] Document Loaders
• [X] Vector Stores / Retrievers
• [ ] Memory
• [X] Agents / Agent Executors
• [X] Tools / Toolkits
• [X] Chains
• [ ] Callbacks/Tracing
• [ ] Async

Reproduction

llm=OpenAI(model_name = "gpt-3.5-turbo", temperature = 0) # text-davinci-003

db = Chroma(persist_directory="db_publications",embedding_function=embedding)
chain = RetrievalQA.from_chain_type(llm=llm, chain_type="stuff", retriever=db.as_retriever(search_kwargs={"k": 2}))

#custom tool is created using my documents 
custom_tool = [
Tool(
    name="Tax_Tool",
    func=chain.run,
    description="Use this tool to answer tax related queries") ]

# built-in tools are loaded
tools = load_tools(["llm-math"], llm=llm)
# built-in tools are appended to the custom tool to maintain priority
for tool in tools:
    custom_tool.append(tool)


# Finally, let's initialize an agent with the tools, the language model, and the type of agent we want to use.
agent = initialize_agent(
    custom_tool, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True)

# Now let's test it out!
result = agent(
    {"input" : "I am married and filing jointly. How much additional Medicare Tax I have to pay if my net earnings are $300,000?"}
)
    ```

### Expected behavior

←[1m> Entering new AgentExecutor chain...←[0m
←[32;1m←[1;3mI need to use a tax calculator to determine the additional Medicare Tax.
Action: Calculator
Action Input: Input $300,000 and select the option for married filing jointly.←[0m2023-05-18 14:15:59.978 Uncaught app exception
Traceback (most recent call last):
  File "C:\Users\hizafa.nadeem\AppData\Local\anaconda3\envs\tax_app\Lib\site-packages\langchain\chains\llm_math\base.py", line 80, in _evaluate_expression
    numexpr.evaluate(
  File "C:\Users\hizafa.nadeem\AppData\Local\anaconda3\envs\tax_app\Lib\site-packages\numexpr\necompiler.py", line 817, in evaluate
    _names_cache[expr_key] = getExprNames(ex, context)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\hizafa.nadeem\AppData\Local\anaconda3\envs\tax_app\Lib\site-packages\numexpr\necompiler.py", line 704, in getExprNames
    ex = stringToExpression(text, {}, context)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\hizafa.nadeem\AppData\Local\anaconda3\envs\tax_app\Lib\site-packages\numexpr\necompiler.py", line 274, in stringToExpression
    c = compile(s, '<expr>', 'eval', flags)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<expr>", line 1
    None (this is not a math problem)
          ^^^^^^^^^^^^^^^^^^
SyntaxError: invalid syntax. Perhaps you forgot a comma?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\hizafa.nadeem\AppData\Local\anaconda3\envs\tax_app\Lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 565, in _run_script
    exec(code, module.__dict__)
  File "D:\Projects\TaxAdvisor\TaxAdviorWithNewDb\agent_gpt_without_observations.py", line 159, in <module>
    result = agent(
             ^^^^^^
  File "C:\Users\hizafa.nadeem\AppData\Local\anaconda3\envs\tax_app\Lib\site-packages\langchain\chains\base.py", line 140, in __call__
    raise e
  File "C:\Users\hizafa.nadeem\AppData\Local\anaconda3\envs\tax_app\Lib\site-packages\langchain\chains\base.py", line 134, in __call__
    self._call(inputs, run_manager=run_manager)
  File "C:\Users\hizafa.nadeem\AppData\Local\anaconda3\envs\tax_app\Lib\site-packages\langchain\agents\agent.py", line 922, in _call
    next_step_output = self._take_next_step(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\hizafa.nadeem\AppData\Local\anaconda3\envs\tax_app\Lib\site-packages\langchain\agents\agent.py", line 800, in _take_next_step
    observation = tool.run(
                  ^^^^^^^^^
  File "C:\Users\hizafa.nadeem\AppData\Local\anaconda3\envs\tax_app\Lib\site-packages\langchain\tools\base.py", line 255, in run
    raise e
  File "C:\Users\hizafa.nadeem\AppData\Local\anaconda3\envs\tax_app\Lib\site-packages\langchain\tools\base.py", line 249, in run
    self._run(*tool_args, run_manager=run_manager, **tool_kwargs)
  File "C:\Users\hizafa.nadeem\AppData\Local\anaconda3\envs\tax_app\Lib\site-packages\langchain\tools\base.py", line 344, in _run
    self.func(
  File "C:\Users\hizafa.nadeem\AppData\Local\anaconda3\envs\tax_app\Lib\site-packages\langchain\chains\base.py", line 236, in run
    return self(args[0], callbacks=callbacks)[self.output_keys[0]]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\hizafa.nadeem\AppData\Local\anaconda3\envs\tax_app\Lib\site-packages\langchain\chains\base.py", line 140, in __call__
    raise e
  File "C:\Users\hizafa.nadeem\AppData\Local\anaconda3\envs\tax_app\Lib\site-packages\langchain\chains\base.py", line 134, in __call__
    self._call(inputs, run_manager=run_manager)
  File "C:\Users\hizafa.nadeem\AppData\Local\anaconda3\envs\tax_app\Lib\site-packages\langchain\chains\llm_math\base.py", line 149, in _call
    return self._process_llm_result(llm_output, _run_manager)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\hizafa.nadeem\AppData\Local\anaconda3\envs\tax_app\Lib\site-packages\langchain\chains\llm_math\base.py", line 103, in _process_llm_result
    output = self._evaluate_expression(expression)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\hizafa.nadeem\AppData\Local\anaconda3\envs\tax_app\Lib\site-packages\langchain\chains\llm_math\base.py", line 87, in _evaluate_expression
    raise ValueError(
ValueError: LLMMathChain._evaluate("
None (this is not a math problem)
") raised error: invalid syntax. Perhaps you forgot a comma? (<expr>, line 1). Please try again with a valid numerical expression

[imeckr]

You can improve the description of the Tax_tool, so the prompting to identify the right tool would be much better by the llm. Currently llm is approximating
I need to use a tax calculator to determine the additional Medicare Tax to be solved by calculator and not tax tool first.

[dosubot[bot]]

Hi, @Hizafa-Nadeem! I'm Dosu, and I'm here to help the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, the issue you raised is about the llm-math tool receiving a word problem instead of a numerical expression. imeckr suggested improving the description of the Tax_tool to prompt users to identify the right tool. This would help avoid using the llm tool for tax calculation problems instead of a tax tool.

Before we close this issue, we wanted to check if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your contribution to the LangChain repository!

[Adesoji1]

same here too