feat: Add search query configuration for multiple searching query handling

[hurxxxx]

Pull Request Checklist

Note to first-time contributors: Please open a discussion post in Discussions and describe your changes before submitting a pull request.

Before submitting, make sure you've checked the following:

• [x] Target branch: Please verify that the pull request targets the dev branch.
• [x] Description: Provide a concise description of the changes made in this pull request.
• [x] Changelog: Ensure a changelog entry following the format of Keep a Changelog is added at the bottom of the PR description.
• [ ] Documentation: Have you updated relevant documentation Open WebUI Docs, or other documentation sources?
• [ ] Dependencies: Are there any new dependencies? Have you updated the dependency versions in the documentation?
• [ ] Testing: Have you written and run sufficient tests for validating the changes?
• [x] Code review: Have you performed a self-review of your code, addressing any coding standard issues and ensuring adherence to the project's coding standards?
• [x] Prefix: To cleary categorize this pull request, prefix the pull request title, using one of the following:
  • BREAKING CHANGE: Significant changes that may affect compatibility
  • build: Changes that affect the build system or external dependencies
  • ci: Changes to our continuous integration processes or workflows
  • chore: Refactor, cleanup, or other non-functional code changes
  • docs: Documentation update or addition
  • feat: Introduces a new feature or enhancement to the codebase
  • fix: Bug fix or error correction
  • i18n: Internationalization or localization changes
  • perf: Performance improvement
  • refactor: Code restructuring for better maintainability, readability, or scalability
  • style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc.)
  • test: Adding missing tests or correcting existing tests
  • WIP: Work in progress, a temporary label for incomplete or ongoing work

Changelog Entry

Description

Currently, web search only processes a single hardcoded query, even when multiple search queries are generated. To improve this functionality, the following changes have been introduced:

• Duplicate keywords are removed before performing the search.
• Added an "Allow Multiple Search Queries" option under Admin Settings → Interface.
  • When set to False, the behavior remains unchanged: only the first query generated by the Query Generation Prompt is searched and added to the context.
  • When set to True, all generated search queries are executed and included in the context.

Added

Changed

• backend/open_webui/config.py
• backend/open_webui/main.py
• backend/open_webui/routers/tasks.py
• backend/open_webui/utils/middleware.py
• src/lib/components/admin/Settings/Interface.svelte
• Changes apply to all translation files located under src/lib/i18n/locales/

Deprecated

Removed

Fixed

Security

Breaking Changes

---

Additional Information

The changes are as follows:

• Removed duplicate keywords
• Added code related to the "Allow Multiple Search Queries" option
• Added a for loop

async def chat_web_search_handler(
    request: Request, form_data: dict, extra_params: dict, user
):

...

queries = queries.get("queries", [])
queries = list(dict.fromkeys(queries))   #Remove duplicate keywords


#searchQuery = queries[0]  ##Disable


 searchQueries = queries
 if not request.app.state.config.ALLOW_MULTIPLE_SEARCH_QUERIES:
     searchQueries = [queries[0]]

 for searchQuery in searchQueries:

...

Screenshots or Videos

[jannikstdl]

Great work! I was wondering if this would also be a thing to apply to RAG search queries. For example, when the Query Generation Prompt is enabled, it generates multiple vector requests, but these don't appear as a status in the UI.

[hurxxxx]

@jannikstdl Are you suggesting that it would be good to display the query keywords in the UI when using RAG search? If so, it seems feasible if the admin approves."

[jannikstdl]

@jannikstdl Are you suggesting that it would be good to display the query keywords in the UI when using RAG search? If so, it seems feasible if the admin approves."

Yes, ecaxtly But i think the RAG query generation is handeled different than the search generation in the backend

[reecelikesramen]

When I wrote #7948 I didn't have the RAG queries in mind but I think it's a good idea to support multiple relevant RAG queries in a single iteration. If you ask an involved question, generating a single query either: A) does not consider each part of the question and retrieves limited information; or B) generates one convoluted query that matches fewer relevant parts. This sort of applies to both a web search and RAG queries in that way.

In a perfect world, I wish that the number of queries could be dynamic in a range. I.e., generate [1..n] questions and the LLM can increase n based on context, but I had trouble getting my weaker LLM (LLaMa 3.3 70B) query generator to do this reliably even after several prompt iterations.

TLDR I think RAG queries should also follow a similar paradigm, although idk how they work in the backend and that may belong in a different PR since it could have different implications for including it.

[hurxxxx]

@reecelikesramen The current RAG queries work very similarly to what you described. Below is the DEFAULT_QUERY_GENERATION_PROMPT_TEMPLATE used for extracting queries for both web search and RAG search. Both types of searches utilize the same template.

### Task:
Analyze the chat history to determine the necessity of generating search queries, in the given language. By default, **prioritize generating 1-3 broad and relevant search queries** unless it is absolutely certain that no additional information is required. The aim is to retrieve comprehensive, updated, and valuable information even with minimal uncertainty. If no search is unequivocally needed, return an empty list.

### Guidelines:
- Respond **EXCLUSIVELY** with a JSON object. Any form of extra commentary, explanation, or additional text is strictly prohibited.
- When generating search queries, respond in the format: { "queries": ["query1", "query2"] }, ensuring each query is distinct, concise, and relevant to the topic.
- If and only if it is entirely certain that no useful results can be retrieved by a search, return: { "queries": [] }.
- Err on the side of suggesting search queries if there is **any chance** they might provide useful or updated information.
- Be concise and focused on composing high-quality search queries, avoiding unnecessary elaboration, commentary, or assumptions.
- Today's date is: {{CURRENT_DATE}}.
- Always prioritize providing actionable and broad queries that maximize informational coverage.

### Output:
Strictly return in JSON format: 
{
  "queries": ["query1", "query2"]
}

### Chat History:
<chat_history>
{{MESSAGES:END:6}}
</chat_history>

You can modify this prompt to fit your requirements (keeping the Guidelines and Output unchanged) and apply the updated prompt in Admin Settings → Interface → Query Generation Prompt to test RAG search functionality.

[Classic298]

any update on this?

[tjbck]

Made it the default behaviour with 9ca47275730487351deca40fd7d31dc8600d92ca after a careful consideration, Thanks for the effort everyone!