feat: Additional (Optional) configuration for external document loader

[TheDropZone]

Pull Request Checklist

Before submitting, make sure you've checked the following:

• [x] Target branch: Verify that the pull request targets the dev branch. Not targeting the dev branch will lead to immediate closure of the PR.
• [x] Description: Provide a concise description of the changes made in this pull request down below.
• [x] Changelog: Ensure a changelog entry following the format of Keep a Changelog is added at the bottom of the PR description.
• [x] Documentation: If necessary, update relevant documentation Open WebUI Docs like environment variables, the tutorials, or other documentation sources.
• [x] Dependencies: Are there any new dependencies? Have you updated the dependency versions in the documentation?
• [x] Testing: Perform manual tests to verify the implemented fix/feature works as intended AND does not break any other functionality. Take this as an opportunity to make screenshots of the feature/fix and include it in the PR description.
• [x] Agentic AI Code: Confirm this Pull Request is not written by any AI Agent or has at least gone through additional human review AND manual testing. If any AI Agent is the co-author of this PR, it may lead to immediate closure of the PR.
• [x] Code review: Have you performed a self-review of your code, addressing any coding standard issues and ensuring adherence to the project's coding standards?
• [x] Title Prefix: To clearly categorize this pull request, prefix the pull request title using one of the following:
  • BREAKING CHANGE: Significant changes that may affect compatibility
  • build: Changes that affect the build system or external dependencies
  • ci: Changes to our continuous integration processes or workflows
  • chore: Refactor, cleanup, or other non-functional code changes
  • docs: Documentation update or addition
  • feat: Introduces a new feature or enhancement to the codebase
  • fix: Bug fix or error correction
  • i18n: Internationalization or localization changes
  • perf: Performance improvement
  • refactor: Code restructuring for better maintainability, readability, or scalability
  • style: Changes that do not affect the meaning of the code (white space, formatting, missing semi-colons, etc.)
  • test: Adding missing tests or correcting existing tests
  • WIP: Work in progress, a temporary label for incomplete or ongoing work

Changelog Entry

Description

Currently, the Document > Content Extraction Engine > External option only enables configuring URL and API key. It uses hard coded values for http method, endpoint, request format, content path, etc. This makes it almost impossible to actually use with a generic external API.

To make this more powerful for a given generic document processing api, the external document loader can be enabled with default settings (ensuring backwards compatibility), or fully configured with the following settings

• url + api key
• http_method: HTTP method to use (PUT, POST, PATCH)
• endpoint: API endpoint path
• request_format: Format for sending the file (binary, form-data, json-base64)

and more advanced settings

• file_field_name: Name of the file field in form-data or JSON
• filename_field_name: Name of the filename field (optional)
• params: Additional parameters to send with the request
• query_params: Query parameters to append to URL
• custom_headers: Custom headers to include in request
• response_content_path: Dot notation path to content in response
• response_metadata_path: Dot notation path to metadata in response
• response_type: Type of response (object, array, text)

This optional configuration enables you to point open-webui at almost any document processing api out there, including custom ones. This enables quick and easy testing of open source document projects, or the ability to easily integrate your own custom api with more configuration.

Added

• Optional (and backwards compatibility) configuration for the External document engine

---

Additional Information

• I tested this configuration using the default options as well as customized for a docstrange API (https://github.com/NanoNets/docstrange), both working great.
• Documentation: Documentation for the new environment variables are being added in a related MR to the docs project: https://github.com/open-webui/docs/pull/812

Screenshots or Videos

External Engine - Default Settings (maintains backwards compat) External Engine - Custom Config (example shows docstrange api) External Engine - Advanced Settings (example shows docstrange api)

Contributor License Agreement

By submitting this pull request, I confirm that I have read and fully agree to the Contributor License Agreement (CLA), and I am providing my contributions under its terms.

[!NOTE] Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in.

[athoik]

@TheDropZone first of all it's nice that you are trying to improve OI.

Although I was under the impression that External Document Loader was design with simplicity in mind. Instead of trying to support all the edge cases in OI, all the "burden" must be handed in the external document loader API.

Also old PUT operation, by default had no specific timeout. It was waiting until server responds.

response = requests.put(f"{url}/process", data=data, headers=headers)

Currently, altough there is a ton of parameters and the timeout is harcoded to 300 seconds / 5 min.

I don't see any real benefit from that PR.

Anyway it's up to author, to reject or approve the PR, just my 0.5 cent.

[TheDropZone]

Yah, that's definitely a valid point. Keep it simple and then require the external api to conform to the demands.

I think I was coming at it from the standpoint of, many users are deploying open-webui alongside other open source projects, and often have the desire to tie them together through config over code when possible. This gives the ability to optionally connect things up without having to update code in either project.

But, at the cost of some extra complexity in the open-webui config for sure. Definitely see both sides.