higress icon indicating copy to clipboard operation
higress copied to clipboard

Support modifying model name based on model mapping when falling back to the target service

Open johnlanni opened this issue 9 months ago • 0 comments

When using the fallback capability, it is similar to the situation in #1901 , where model mapping cannot be set.

Image

The configuration method also needs to be modified in a similar way.

Image

Can be changed to the following configuration:

apiVersion: extensions.higress.io/v1alpha1
kind: WasmPlugin
metadata:
  name: model-mapper.internal
  namespace: higress-system
spec:
  defaultConfigDisable: true
  failStrategy: FAIL_OPEN
  matchRules:
  - config:
      modelMapping:
        'gpt-4-*': "qwen-max"
        'gpt-4o': "qwen-vl-plus"
        '*': "qwen-turbo"
    configDisable: false
    # This configuration takes effect when the following ingress and service match simultaneously.
    ingress:
    - ai-route-dashscope.internal
    service:
    - llm-dashscope.internal.dns
  - config:
      modelMapping:
        'gpt-4-*': "qwen-plus"
        '*': "qwen-long"
    configDisable: false
    # This configuration takes effect when the following ingress and service match simultaneously.
    ingress:
    # When fallback is triggered, it will go through this ingress.
    - ai-route-dashscope.fallback.internal
    service:
    - llm-dashscope.internal.dns    
  phase: AUTHN
  priority: 800
  url: oci://higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/model-mapper:1.0.0

johnlanni avatar Mar 15 '25 07:03 johnlanni