Add optional APIM (API Management) to infrastructure

[pamelafox]

We're discussing APIM integration in this sample, as we know many developers are manually adding APIM, so we'd like to make it easier for everyone.

However, there are many ways to use APIM, so we want to clarify what is most popular with developers.

If you're interested in APIM, please clarify:

• What are your goals in using APIM?
  • Monitoring token usage?
  • Enforcing authentication?
  • Rate limiting usage? (B2B or B2C)
  • Load balancing between instances?
• Do you want to put APIM in front of the entire backend (/chat, /ask, /content) or just the OpenAI endpoint itself?

We'd love feedback, thanks!

[vrajroutu]

The load balancing between instances is performing exceptionally well, even with over 3000+ users. I implemented this solution using the guidelines provided in the reference article:

Azure OpenAI Load Balancing.

[jws-m-gotoh]

Thanks for the great code. In my case, I want to enforce authentication using APIM. APIM is placed in front of the OpenAI endpoint.

[UmarMohamedUsman]

@pamelafox, all the uses cases you mentioned are really important. My priority order :)

1. Load balancing across multiple AOAI instances (very critical)
2. Rate limiting
3. Enforcing authentication
4. Monitoring token usage

APIM in front of AOAI endpoint should be good enough.

[adamhrin]

Hello, thanks for the repo, we appreciate the effort!

We are interested to put this into production and put APIM in front of the entire backend. Could you point us to what changes would need to be done in the code to enable APIM for authentication (via Azure AD)?

Is it possible to enable authentication through APIM, and if so how do we need to configure the authentication setup compared to your instruction with Azure AD apps? We are interested in this so that we can for example restrict IP addresses and enable certificates.

Thank you for your help!