[Question]: When using the agent, I choose the LLM component, which allows me to set the model to responsive_format={'type ':'json object'}

[wuqilei-code]

Self Checks

• [x] I have searched for existing issues search for existing issues, including closed ones.
• [x] I confirm that I am using English to submit this report (Language Policy).
• [x] Non-english title submitions will be closed directly ( 非英文标题的提交将会被直接关闭 ) (Language Policy).
• [x] Please do not modify this template :) and fill in all the required fields.

Describe your problem

I am currently using the LLM component in the agent, and I need to ask the big model to help me generate JSON structured data through prompt words. However, the current generation is not accurate. I see that the big model has a parameter that can boot JSON data support: response_format={ 'type': 'json_object' } How can I make it add or modify parameters when the agent calls the large model

[dosubot[bot]]

To enable the LLM component to generate accurate JSON-structured data using the response_format parameter, you’ll need to update the code so that this parameter is included in the configuration passed to the LLM. Currently, the GenerateParam class and its gen_conf method only support a fixed set of parameters (like max_tokens, temperature, etc.), and do not allow arbitrary parameters such as response_format to be passed through by default.

The LLM client does accept extra parameters via the gen_conf dictionary, so you can extend the GenerateParam class and its gen_conf method to include response_format (or any other custom parameter you need). For example, you could add a new attribute for response_format to GenerateParam, update gen_conf to include it if set, and ensure it is passed along to the LLM invocation. This will allow the LLM to receive and honor the response_format={'type': 'json_object'} setting for structured JSON output.

If you want to make this more flexible, you could also modify the parameter handling to allow arbitrary key-value pairs to be merged into gen_conf, so you’re not limited to a fixed schema. This would let you pass any supported LLM parameter through the agent interface without further code changes in the future.

There isn’t existing documentation or issues that cover this specific configuration, so code changes are required to support this use case. The relevant logic is in agent/component/generate.py and rag/llm/chat_model.py, where gen_conf is constructed and passed to the LLM client as keyword arguments, making it straightforward to extend for new parameters like response_format.
See supporting code details

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

[KevinHuSh]

We're working on that.