web-ui icon indicating copy to clipboard operation
web-ui copied to clipboard

Add rate limiting configuration for LLM providers

Open hoangnb24 opened this issue 9 months ago • 1 comments

  • Introduce rate limit parameters (requests/sec and max bucket size) for all LLM providers
  • Update webui.py to include new rate limit inputs in UI and function signatures
  • Modify utils.py to create InMemoryRateLimiter for each LLM provider
  • Add rate limiter configuration to all supported LLM models

hoangnb24 avatar Feb 12 '25 04:02 hoangnb24

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Feb 12 '25 04:02 CLAassistant