openlimit
openlimit copied to clipboard
shobrook
Maximize your usage of OpenAI models without hitting rate limits
For example 40k TPM is converted to 666.66 TPS I've a summarization usecase where the incoming token is 800+, in which case the logic seems to wait indefinitely. Has anyone...
- Use latest redis version since aioredis is depreciated - Add simple implementation of wait_for_capacity_sync to use async and wait
Hello, The package does not work on Python >=3.11 ``` >>> from openlimit import EmbeddingRateLimiter Traceback (most recent call last): File "", line 1, in File "lib/python3.11/site-packages/openlimit/__init__.py", line 1, in...
aioredis has been [deprecated](https://github.com/aio-libs-abandoned/aioredis-py#-aioredis-is-now-in-redis-py-420rc1-) as it is now included in the official redis Python package. As aioredis [fails to import](https://github.com/aio-libs-abandoned/aioredis-py/issues/1409) on Python 3.11+, it prevents me from using openlimit, even...
Issue Description Problem: When using the new OpenAI library in my asynchronous application, I've encountered an issue related to rate limiting. Description: It appears that when I run my asynchronous...
Following (v0.3.0) ```python #!/usr/bin/python3 import sys import traceback import openai from openlimit import ChatRateLimiterWithRedis openai.api_key = "" request_timeout = 3 rate_limiter = ChatRateLimiterWithRedis( request_limit=3500, token_limit=90000, redis_url="redis://localhost:6379/0" ) @rate_limiter.is_limited() def call_openai(**chat_params):...
``` rate_limiter = ChatRateLimiterWithRedis( request_limit=1, token_limit=40000, redis_url="redis://localhost:6379" ) chat_params = { "model": "gpt-4", "messages": [{"role": "user", "content": "Hello!"}], } with rate_limiter.limit(**chat_params): response = openai.ChatCompletion.create(**chat_params) ``` The code above is hanging...
Hi, thanks for the implementation! We were planning to add this feature in zeno-build library as well ([#132](https://github.com/zeno-ml/zeno-build/issues/132)). I was curious how well do the OpenAI servers work in practice...
Hi there, can you support me to run this with https://pypi.org/project/pyautogen/ ?
