litellm icon indicating copy to clipboard operation
litellm copied to clipboard
Published 1 week ago • verified publisher icon BerriAI

[22/01/2024 - 29/01/2024] New Models/Endpoints/Providers/Improvements

Open ishaan-jaff opened this issue 1 year ago • 1 comments

Budget Management

  • [ ] Add testing for proxy_server.track_cost_callback - writing to SpendLogs Table
  • [ ] Allow Users to Set Budgets per user per month https://github.com/BerriAI/litellm/issues/1544
  • [ ] See (By User) spend this month https://github.com/BerriAI/litellm/issues/1545
  • [ ] See (By User) to see total requests, Total tokens used (completion, embedding), spend,
  • [ ] Update Budgets per user (allow users to reset budgets) https://github.com/BerriAI/litellm/issues/1546
  • [x] Allow setting budgets per key https://github.com/BerriAI/litellm/issues/1548
  • [x] Allow setting tpm/rpm limits per key https://github.com/BerriAI/litellm/issues/1550

Spend Tracking

  • [x] Support input/output cost per second - custom pricing https://github.com/BerriAI/litellm/issues/1374
  • [x] Emit spend as part of logging call https://github.com/BerriAI/litellm/issues/1547
  • [ ] Log token usage in spend logs https://github.com/BerriAI/litellm/issues/1549
  • [ ] Support bedrock region pricing in completion_cost - https://github.com/BerriAI/litellm/issues/1565
  • [ ] Completion cost support for image gen models - https://github.com/BerriAI/litellm/issues/1558
  • [ ] add Region-specific amazon text embedding pricing - https://github.com/BerriAI/litellm/issues/1566
  • [ ] Allow local model cost maps for proxy https://github.com/BerriAI/litellm/issues/1434

Misc

  • [x] real Sagemaker streaming support - https://github.com/BerriAI/litellm/issues/1384
  • [ ] Gemini Pro - VertexAI function calling when streaming support - https://github.com/BerriAI/litellm/issues/1263
  • [x] Allow users to add their own logic to /key/generate https://github.com/BerriAI/litellm/issues/1462
  • [ ] Caching improvement for embedding calls - https://github.com/BerriAI/litellm/issues/1350
  • [ ] Make Authentication Errors OpenAI compatible: https://github.com/BerriAI/litellm/issues/1554

ishaan-jaff avatar Jan 22 '24 15:01 ishaan-jaff

Known Backlog:

  1. OpenMeter spend tracking support - https://github.com/BerriAI/litellm/issues/1268
  2. tracking - https://github.com/BerriAI/litellm/issues/997
  3. tracking - https://github.com/BerriAI/litellm/issues/1188 cc: @r1cc4rd0m4zz4
  4. tracking - https://github.com/BerriAI/litellm/issues/1192
  5. yandexgpt support requested by 2 users - https://github.com/BerriAI/litellm/issues/1254
  6. Aphrodite engine endpoint - https://github.com/PygmalionAI/aphrodite-engine, https://github.com/BerriAI/litellm/pull/1153
  7. TheBloke/NexusRaven-V2-13B-GGUF function calling support https://github.com/BerriAI/litellm/issues/1060 cc: @homanp
  8. Predibase lorax support - [Feature]: Add support for Predibase Lorax apis #1253
  9. Deepsparse CPU inference support - https://github.com/neuralmagic/deepsparse/issues/1452
  10. Consistent format for model prices and context window - https://github.com/BerriAI/litellm/issues/1375
  11. Vertex AI model garden support - https://github.com/BerriAI/litellm/issues/920

krrishdholakia avatar Jan 22 '24 15:01 krrishdholakia

New one here: https://github.com/BerriAI/litellm/issues/1665

ishaan-jaff avatar Jan 29 '24 19:01 ishaan-jaff