upgrade pydantic to 2.*

[LeoQuote]

Description

pydantic 2 introduced multiple enhancement, faster performance, and it was released 9 months ago, it's time to move forward and get better performance, more features.

summary of the changes:

1. upgrade langchain, inference-client qdrant-client to the latest, add pydantic-extra-types as it's a new package now
2. apply changes due to pydantic breaking changes, using bump-pydantic api
3. fix a few type errors brought by YAML parse, those errors should exist before the upgrade, the strong type check by pydantic 2 exposed them.
4. rename model_config field in AdvancedChatMessageEntity, MultiDatasetRouterAgent, AgentConfiguration to ai_model_config because model_config is reserved by pydantic 2

This is a big update and could be harmful, I advise someone else set up a coverage report like codecov, then I can see which part of the code is not covered by tests and add more tests to bring us more confidence.

Type of Change

Please delete options that are not relevant.

• [x] Dependency upgrade

How Has This Been Tested?

• [x] Start a server that works

@LeoQuote ➜ /workspaces/dify/api (upgrade_pydantic) $ FLASK_APP=app.py flask run 
/home/vscode/.local/lib/python3.10/site-packages/pydantic/_internal/_fields.py:151: UserWarning: Field "model_type" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
/home/vscode/.local/lib/python3.10/site-packages/pydub/utils.py:170: RuntimeWarning: Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work
  warn("Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work", RuntimeWarning)
/home/vscode/.local/lib/python3.10/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.llms import Tongyi`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/home/vscode/.local/lib/python3.10/site-packages/pydantic/_internal/_fields.py:151: UserWarning: Field "model_type_instance" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/home/vscode/.local/lib/python3.10/site-packages/pydantic/_internal/_fields.py:151: UserWarning: Field "model_schema" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
WARNING:root:Missing subclass of ExternalDataTool in /workspaces/dify/api/core/external_data_tool/api/api.py, Skip.
/home/vscode/.local/lib/python3.10/site-packages/pydantic/_internal/_fields.py:186: UserWarning: Field name "raise_error" shadows an attribute in parent "BaseCallbackHandler"; 
  warnings.warn(
/home/vscode/.local/lib/python3.10/site-packages/pydantic/_internal/_fields.py:186: UserWarning: Field name "run_inline" shadows an attribute in parent "BaseCallbackHandler"; 
  warnings.warn(
/home/vscode/.local/lib/python3.10/site-packages/pydantic/_internal/_fields.py:151: UserWarning: Field "model_configuration" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/home/vscode/.local/lib/python3.10/site-packages/langchain/__init__.py:29: UserWarning: Importing BasePromptTemplate from langchain root module is no longer supported. Please use langchain_core.prompts.BasePromptTemplate instead.
  warnings.warn(
/home/vscode/.local/lib/python3.10/site-packages/langchain/__init__.py:29: UserWarning: Importing PromptTemplate from langchain root module is no longer supported. Please use langchain_core.prompts.PromptTemplate instead.
  warnings.warn(
/home/vscode/.local/lib/python3.10/site-packages/langchain/__init__.py:29: UserWarning: Importing LLMChain from langchain root module is no longer supported. Please use langchain.chains.LLMChain instead.
  warnings.warn(
/home/vscode/.local/lib/python3.10/site-packages/pydantic/_internal/_fields.py:151: UserWarning: Field "model_credential_schema" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/home/vscode/.local/lib/python3.10/site-packages/pydantic/_internal/_fields.py:151: UserWarning: Field "model_type" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
 * Serving Flask app 'app.py'
 * Debug mode: off
INFO:werkzeug:WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:5000
INFO:werkzeug:Press CTRL+C to quit

Suggested Checklist:

• [x] I have performed a self-review of my own code
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] My changes generate no new warnings
• [x] I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods
• [ ] optional I have made corresponding changes to the documentation
• [ ] optional I have added tests that prove my fix is effective or that my feature works
• [ ] optional New and existing unit tests pass locally with my changes

[takatost]

We refactored most of the code in the workflow version, this PR will be reviewed after the workflow version is released

[LeoQuote]

https://github.com/langchain-ai/langchain/blob/a27d88f12aea9ca13e4c4e72f2eceb4cc6cc090f/libs/core/langchain_core/pydantic_v1/init.py

unfortunately langchain is not fully compatible with pydantic v2, they are using a kind of hack right now, causing some compatible issue if we upgrade to pydantic v2, including :

RuntimeError: no validator found for <class 'core.app.entities.app_invoke_entities.ModelConfigWithCredentialsEntity'>, see `arbitrary_types_allowed` in Config

This issue can be resolved with

from langchain_core.pydantic_v1 import BaseModel

and using everything langchain provided, but this solution is hack and problematic, so I'm giving up and wait for langchain to resolve those issues.