Allow costs and window size limits to be specified in the config_list

[afourney]

Why are these changes needed?

See issue #1680.

TL;DR: Token limits and costs are hardcoded in various files. Adding new models requires editing numerous files. Costs are keyed off model names, but those name may not be reliable. For example, "gpt-4" on Azure might point to an 8k, 32k, or 128k model. There is no requirement that these match with OpenAI names. Likewise, model names may not match at all (e.g., "my-gpt-4") Adding new models or providers requires hand-editing core library files.

This PR allows costs and limits to be specified in the config. E.g.,

[
        "model": "gpt-4-turbo",
        "api_key": "blahblahblah",
        "base_url": "https://mymodel.openai.azure.com/",
        "api_type": "azure",
        "api_version": "2023-12-01-preview",
        "window_size":  { "input": 128000, "output":  4096 },
        "token_cost_1k":  { "input": 0.01, "output": 0.03 ) 
]

Limitations: Vision models are not fully-supported. We can compute the cost only after we get the tokens counts back after making the call.

Related issue number

Closes #1680, #1270

Checks

• [ ] I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• [ ] I've added tests (if relevant) corresponding to the changes introduced in this PR.
• [ ] I've made sure all auto checks have passed.

[codecov-commenter]

Codecov Report

Attention: 12 lines in your changes are missing coverage. Please review.

Comparison is base (f68c09b) 37.85% compared to head (7148a4d) 49.60%.

Files	Patch %	Lines
autogen/oai/client.py	25.00%	12 Missing :warning:

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1682       +/-   ##
===========================================
+ Coverage   37.85%   49.60%   +11.75%     
===========================================
  Files          50       50               
  Lines        5733     5737        +4     
  Branches     1297     1408      +111     
===========================================
+ Hits         2170     2846      +676     
+ Misses       3391     2662      -729     
- Partials      172      229       +57     

Flag	Coverage Δ
unittests	49.57% <25.00%> (+11.72%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[afourney]

wondering if this needs to be passed into cost or if we can just pass all the args into the client constructor (as I believe is already done?) and re-use in different methods, that way we don't have to change the API every time we want to use a new config value

I'd love to figure this out. I think the issue I was working around is that the main wrapper client takes the config list, which may create many OAI or Azure clients, and we don't know which is used until we make the call. The cost method I had initially targeted for modification is a member of the wrapper client class.

I'm not sure I fully understand the solution to this you are proposing, but I would be happy to get rid of the new parameter if possible.

[olgavrou]

wondering if this needs to be passed into cost or if we can just pass all the args into the client constructor (as I believe is already done?) and re-use in different methods, that way we don't have to change the API every time we want to use a new config value

I'd love to figure this out. I think the issue I was working around is that the main wrapper client takes the config list, which may create many OAI or Azure clients, and we don't know which is used until we make the call. The cost method I had initially targeted for modification is a member of the wrapper client class.

I'm not sure I fully understand the solution to this you are proposing, but I would be happy to get rid of the new parameter if possible.

If we pass any extra kwargs that we need to the client constructors so for example OpenAIClient, then when OpenAIClient's cost or create or other function is called it will have access to things like token cost, window size, etc

[yiranwu0]

Should we force or at least warn user to pass in token_cost if they are using azure?

From my previous experience azure always return 'gpt-4' in response['model'], this will very likely cause a wrong calculation of cost. See #1270

[afourney]

Should we force or at least warn user to pass in token_cost if they are using azure?

From my previous experience azure always return 'gpt-4' in response['model'], this will very likely cause a wrong calculation of cost. See #1270

That is a good idea. This has been my experience too. Perhaps it should warn or force for any model that isn't OpenAI.

[gitguardian[bot]]

⚠️ GitGuardian has uncovered 96 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard. Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
12853598	Triggered	Generic High Entropy Secret	79dbb7bc2561713bc11225849e408dc74db1228f	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	e43a86c78f3f947b6e142b3aaf36e7a9852f7078	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	bdb40d77d7be9ea42e5fa28f6e851edf549ef0af	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	954ca451f949a4924578bb7e3e93c97ad4ba1dd5	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	79dbb7bc2561713bc11225849e408dc74db1228f	test/oai/test_utils.py	View secret
10404662	Triggered	Generic CLI Secret	eff19acf1365e34fe17d9ac0939666f32b3ceda5	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	06a0a5ddb39fb3000e40cb4872741dead9fcb7e0	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	0524c774dda696855e9a8f87fe0b9ed96cce13f6	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	d7ea410501cb96bc97a203a0a95431541515f9cc	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	e43a86c78f3f947b6e142b3aaf36e7a9852f7078	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	841ed315e2f19d79a6b86ed587eb6e0fc4a0c0da	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	802f099588bedf1d022b2bba5fb534635df8e6f1	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	9a484d8589761d58940e3c0b79215204ce6a23c1	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	e973ac38ea4f7b36687ea03aa44f770d7e2ddcac	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	89650e74f572bbc60e9c24e04b4c601616e439c7	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	e07b06bc939353dee2afc6dd52c159d6cb3428d5	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	abe4c419c45a244e2399508f9ffef606ce6a4685	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	7362fb9cf4f0a6b3e7ae4e97229afad655d5d87b	.github/workflows/dotnet-release.yml	View secret
12853599	Triggered	Generic High Entropy Secret	79dbb7bc2561713bc11225849e408dc74db1228f	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	e43a86c78f3f947b6e142b3aaf36e7a9852f7078	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	954ca451f949a4924578bb7e3e93c97ad4ba1dd5	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	bdb40d77d7be9ea42e5fa28f6e851edf549ef0af	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	abad9ff4487444324d1916a6b94c8049b4cab9e7	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	954ca451f949a4924578bb7e3e93c97ad4ba1dd5	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	c7bb588a684a038dbee7f9dc9afd6d44ce35ac3a	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	b97b99d4b2cfe735a7aa46258508d4a7cda9cad5	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	e43a86c78f3f947b6e142b3aaf36e7a9852f7078	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	79dbb7bc2561713bc11225849e408dc74db1228f	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	79dbb7bc2561713bc11225849e408dc74db1228f	test/oai/test_utils.py	View secret
10493810	Triggered	Generic Password	49e8053dd1e5456d3758b4a85f5721e9c9b12e16	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	501610b4fc0c649fa2b1cbf0fd72fa6f14f026d0	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	49e8053dd1e5456d3758b4a85f5721e9c9b12e16	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	501610b4fc0c649fa2b1cbf0fd72fa6f14f026d0	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	d422c63596fce7b84e0ef8a7bd6518d5f4336eaf	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	97fa339ac749b3c7d51f1b0fae156d41c02b214e	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	49e8053dd1e5456d3758b4a85f5721e9c9b12e16	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	d422c63596fce7b84e0ef8a7bd6518d5f4336eaf	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	97fa339ac749b3c7d51f1b0fae156d41c02b214e	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	d422c63596fce7b84e0ef8a7bd6518d5f4336eaf	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	97fa339ac749b3c7d51f1b0fae156d41c02b214e	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	501610b4fc0c649fa2b1cbf0fd72fa6f14f026d0	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10404696	Triggered	Generic High Entropy Secret	954ca451f949a4924578bb7e3e93c97ad4ba1dd5	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	bdb40d77d7be9ea42e5fa28f6e851edf549ef0af	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	79dbb7bc2561713bc11225849e408dc74db1228f	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	e43a86c78f3f947b6e142b3aaf36e7a9852f7078	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	79dbb7bc2561713bc11225849e408dc74db1228f	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	bdb40d77d7be9ea42e5fa28f6e851edf549ef0af	test/oai/test_utils.py	View secret
12853602	Triggered	Generic High Entropy Secret	79dbb7bc2561713bc11225849e408dc74db1228f	test/oai/test_utils.py	View secret
11616921	Triggered	Generic High Entropy Secret	a86d0fde2e667f9177eea55da17312b770c9d76b	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	394561b4629222c19d8bb3bc58222fc8813a5833	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	3eac646b8974e1d1be3fde557b055563f56f2f5f	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	f45b55337aa7f9d6151f21f2c55ba5dc83f95b79	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	65632487224aae2c7faca67de40fd2bda3ad3905	notebook/agentchat_agentops.ipynb	View secret
12853598	Triggered	Generic High Entropy Secret	2b3a9ae05569d38cfc89d9daa4b1612b8406d178	test/oai/test_utils.py	View secret
12853598	Triggered	Generic High Entropy Secret	c03558f5b2708c31df36666486f54b2714435162	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	c03558f5b2708c31df36666486f54b2714435162	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	2b3a9ae05569d38cfc89d9daa4b1612b8406d178	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	0a3c6c49834b4f68c688c1d2e76a5ebb8f7d91e2	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	76f5f5a66532a3b95d03fe0ae8a56a59e43012e1	test/oai/test_utils.py	View secret
10404662	Triggered	Generic CLI Secret	954ca451f949a4924578bb7e3e93c97ad4ba1dd5	.github/workflows/dotnet-build.yml	View secret
12853599	Triggered	Generic High Entropy Secret	2b3a9ae05569d38cfc89d9daa4b1612b8406d178	test/oai/test_utils.py	View secret
12853599	Triggered	Generic High Entropy Secret	c03558f5b2708c31df36666486f54b2714435162	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	76f5f5a66532a3b95d03fe0ae8a56a59e43012e1	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	0a3c6c49834b4f68c688c1d2e76a5ebb8f7d91e2	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	3b79cc6fdfa485026262ec44aa5f1f0238bf7c3d	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	11baa52155c3846299d39ed0b5a814b33b0eb671	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	c03558f5b2708c31df36666486f54b2714435162	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	2b3a9ae05569d38cfc89d9daa4b1612b8406d178	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	c03558f5b2708c31df36666486f54b2714435162	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	2b3a9ae05569d38cfc89d9daa4b1612b8406d178	test/oai/test_utils.py	View secret
10493810	Triggered	Generic Password	3b79cc6fdfa485026262ec44aa5f1f0238bf7c3d	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	11baa52155c3846299d39ed0b5a814b33b0eb671	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	11baa52155c3846299d39ed0b5a814b33b0eb671	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	3b79cc6fdfa485026262ec44aa5f1f0238bf7c3d	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10404696	Triggered	Generic High Entropy Secret	0a3c6c49834b4f68c688c1d2e76a5ebb8f7d91e2	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	76f5f5a66532a3b95d03fe0ae8a56a59e43012e1	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	c03558f5b2708c31df36666486f54b2714435162	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	2b3a9ae05569d38cfc89d9daa4b1612b8406d178	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	2b3a9ae05569d38cfc89d9daa4b1612b8406d178	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	c03558f5b2708c31df36666486f54b2714435162	test/oai/test_utils.py	View secret

and 16 others.

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secrets safely. Learn here the best practices.
3. Revoke and rotate these secrets.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

[afourney]

Closing this PR, since it is quite dated. Will open a new issue.