wch
Use of Azure endpoints?
This is really awesome. Is it possible to use with Azure endpoints?
https://learn.microsoft.com/en-us/azure/cognitive-services/openai/quickstart?pivots=programming-language-python&tabs=command-line
Thanks
Yes, it should be possible, although I have not tested it with Azure myself.
When you call chatstream.chat_server(), you can pass it a url. See the docs here:
https://wch.github.io/chatstream/reference/chat_server.html#chatstream.chat_server
I believe you should still set the OPENAI_API_KEY environment variable. In the docs you linked to, they recommend setting AZURE_OPENAI_KEY, which you could do, but then you would need to get that env var's value manually and pass it like chatstream.chat_server(api_key=os.getenv("AZURE_OPENAI_KEY"), ...).
If you test it out, please let me know how it goes!
I tried but it didn't seem to work - perhaps due to company firewall. I will dig in further
Hi @wch, any chance you can add proxy and deployment_name as arguments to chat_server? There have been some issues regarding discrepancies between model and engine/deployment names on the Azure endpoint.
I modified my server function in basic/app.py as follows:
def server(input: Inputs, output: Outputs, session: Session):
chatstream.chat_server(
"mychat",
model="Turbo", # also tried "gpt-35-turbo", didn't work
url=os.getenv("OPENAI_API_BASE"),
api_key=os.getenv("OPENAI_API_KEY"),
)
and getting
Unhandled error: 'Could not automatically map Turbo to a tokeniser. Please use
tiktok.get_encodingto explicitly get the tokeniser you expect.'
When I set model="gpt-3.5-turbo", I got
Unhandled error: Please set the OPENAI_API_KEY environment variable to your OpenAI API key. which sounds like it's still using the default Open AI url, not my Azure endpoint.
I'm not sure if we're using langchain but I wanted to include a few issues in case they're relevant:
- https://github.com/openai/openai-python/issues/318
- https://github.com/hwchase17/langchain/issues/3251
- https://github.com/hwchase17/langchain/pull/6697/files
- https://github.com/hwchase17/langchain/issues/4575
- https://github.com/hwchase17/langchain/pull/3076
In case it is useful information gptstudio provides the ability to connect to an Azure endpoint.
https://github.com/MichelNivard/gptstudio
@trangdata I don't have access to Azure OpenAI, but I just applied for access. Hopefully I will be given access in the near future, but until then I'm not able to test anything with Azure.
I'll try adding proxy and deployment_name in a pull request.
For reference, this is their documentation on the differences between using OpenAI's endpoints vs. Azure's endpoints. https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/switching-endpoints
@trangdata I don't see proxy as a possible parameter to .create() or .acreate(). Taking a quick look at the openai source code, I think you will have to set it like this:
import openai
openai.proxy = "..."
https://github.com/openai/openai-python/blob/b82a3f7e4c462a8a10fa445193301a3cefef9a4a/openai/init.py#L56
I've created a PR for Azure support, #10. Please see that PR and test it out. There is more information and instructions in the PR.
Thank you for the quick response! I installed the new chatstream version in the PR, added some openai configs (below) to my app.py, and uses endpoint_type = "azure" in chat_server, and it doesn't error out but the app just kinda stalls with ... in the response text box.
openai.api_type = os.getenv("OPENAI_API_TYPE")
openai.api_base = os.getenv("OPENAI_API_BASE")
openai.api_key = os.getenv("OPENAI_API_KEY")
openai.proxy = os.getenv("OPENAI_PROXY")
openai.api_version = "2023-03-15-preview" # "2022-12-01"
When I set model="Turbo" or model="gpt-35-turbo", which is what I would do for other projects using the Azure endpoint, I received the same error as before.
Unhandled error: 'Could not automatically map gpt-35-turbo to a tokeniser. Please use
tiktok.get_encodingto explicitly get the tokeniser you expect.'
Quick answer because I’m on my phone: shouldn’t there be a period in gpt-3.5-turbo?
Actually no. Azure is very odd with their engine/deployment name and model name. See https://github.com/hwchase17/langchain/issues/1577#issuecomment-1479489910 and example: https://github.com/hwchase17/langchain/issues/3251#issuecomment-1566531223
In the PR I just pushed some code to map from Azure model names to OpenAI model names. @trangdata Please give it another shot. Thanks!
Thanks, Winston!
So this time, instead of erroring out immediately, the app just kinda stalls forever. And when I Ctrl C, I see:
Task exception was never retrieved future: <Task finished name='Task-9' coro=<stream_to_reactive.
.task_main() done, defined at /Users/let20/Documents/chatstream/chatstream/init.py:547> exception=InvalidRequestError(message="Must provide an 'engine' or 'deployment_id' parameter to create a <class 'openai.api_resources.chat_completion.ChatCompletion'>", param='engine', code=None, http_status=None, request_id=None)>
Full message
(chatstream) ➜ chatstream shiny run examples/basic/app.py --launch-browser
INFO: Started server process [19238]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO: 127.0.0.1:58174 - "GET / HTTP/1.1" 200 OK
INFO: ('127.0.0.1', 58175) - "WebSocket /websocket/" [accepted]
INFO: connection open
^CINFO: Shutting down
INFO: connection closed
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [19238]
Task exception was never retrieved
future: <Task finished name='Task-9' coro=<stream_to_reactive.
I received access to Azure OpenAI, and got it working; see #10 again for instructions and an example app. Please give it a shot!
The API in that PR is currently really awkward to accommodate OpenAI and Azure-OpenAI, and I will clean that up before merging.
Hmm, so I installed the new changes, but the app hangs after I enter a prompt.
openai.error.APIConnectionError: Error communicating with OpenAI
Full message
Task exception was never retrieved
future: <Task finished name='Task-9' coro=<stream_to_reactive.
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/Users/let20/Documents/chatstream/chatstream/init.py", line 550, in task_main func = await func # type: ignore ^^^^^^^^^^ File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_resources/chat_completion.py", line 45, in acreate return await super().acreate(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 217, in acreate response, _, api_key = await requestor.arequest( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_requestor.py", line 372, in arequest result = await self.arequest_raw( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/let20/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_requestor.py", line 685, in arequest_raw raise error.APIConnectionError("Error communicating with OpenAI") from e openai.error.APIConnectionError: Error communicating with OpenAI
Is there any way of testing the api call is able to connect?
I am also getting the hanging issue but I get an SSL error saying that SSL:certificate_verify_failed for our Azure endpoint.
Thanks!
@iainmwallace I have a had some difficultly getting error messages to show up immediately. I think it has something to do with it receiving the data on a separate Task, but I'm not 100% sure.
@trangdata I see some references to a proxy in the stack trace, and I know you mentioned something about a proxy earlier. I don't see anything in the Azure OpenAI docs about using a proxy, though. How are you using it?
For both of you, have you tried running the example app in https://github.com/wch/chatstream/pull/10#issuecomment-1641451668 ?
Note that you will need to set the openai.api_base URL (https://winstontest.openai.azure.com in the example) as well as a deployment ID (my-gpt-35-turbo in the example).
Here is a simple script I used to test communication with Azure:
import openai
openai.api_type = "azure"
openai.api_base = "https://winstontest.openai.azure.com/" # <- Customize this
openai.api_version = "2023-03-15-preview"
openai.api_key = "xxxxxxx" # <- Customize this
resp = await openai.ChatCompletion.acreate(
deployment_id="my-gpt-35-turbo", # <- Customize this
messages = [{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":"Tell me about yourself"}],
stream=True
)
print(resp)
async for chunk in resp:
print(chunk)
Note that this must be run with an interpreter that is run using asyncio, like ipython or (I think) Jupyter. If you use the plain python interpreter, it will give this error: SyntaxError: 'await' outside function.
If you run this, it should print out a lot of messages which look something like this:
{
"id": "chatcmpl-7e7JqYddw8TgaKtbzs6VTwqk1rE5R",
"object": "chat.completion.chunk",
"created": 1689795610,
"model": "gpt-35-turbo",
"choices": [
{
"index": 0,
"finish_reason": null,
"delta": {
"content": " efficient"
}
}
],
"usage": null
}
{
"id": "chatcmpl-7e7JqYddw8TgaKtbzs6VTwqk1rE5R",
"object": "chat.completion.chunk",
"created": 1689795610,
"model": "gpt-35-turbo",
"choices": [
{
"index": 0,
"finish_reason": null,
"delta": {
"content": "."
}
}
],
"usage": null
}
{
"id": "chatcmpl-7e7JqYddw8TgaKtbzs6VTwqk1rE5R",
"object": "chat.completion.chunk",
"created": 1689795610,
"model": "gpt-35-turbo",
"choices": [
{
"index": 0,
"finish_reason": "stop",
"delta": {}
}
],
"usage": null
}
If this works, I think the example app in the PR should work.
Thanks, that is incredibly helpful.
I get an errror 'InvalidURL' when I use the function acreate:
resp = await openai.ChatCompletion.acreate(model="gpt-4", deployment_id="gpt-4", messages=[{"role": "user", "content": "Hello world"}])
But with the chatcompletion function example from https://github.com/openai/openai-python#usage, it works as expected:
chat_completion = openai.ChatCompletion.create(model="gpt-4", deployment_id="gpt-4",messages=[{"role": "user", "content": "Hello world"}])
A couple questions:
What version of openai are you using?
>>> import openai
>>> openai.__version__
'0.27.8'
What happens if you call acreate() without model? My understanding is that the model parameter isn't used by Azure, and perhaps including it doesn't cause problems for create(), but does cause problems for acreate(). See here for the docs about it: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/switching-endpoints#keyword-argument-for-model
I am using 0.27.8 and I get the same error when removing the model parameter in acreate(). Removing create() continues to work without the model parameter.
I will ask internally if anyone has gotten this function to work, and will report back
I don't think you're running into this problem, but for future reference, I just realized that I need to mention one very important thing about the acreate() code I provided -- it must be run in an environment that's run with asyncio so that await can be called from the top level. I'll edit the comment above to mention this.
In practice, I ran it with ipython. (I think it should work from a Jupyter notebook as well but I'm not 100% sure.)
If you run it with the plain python interpreter, it will give this error when you use await at the top level. For example:
>>> resp = await openai.ChatCompletion.acreate( .... )
SyntaxError: 'await' outside function
I am trying to use it in conjunction with azure openai... did try passing the params to
chatstream.chat_server( "mychat", url=config['AZURE_OPENAI_BASE_URL'], api_key=config['AZURE_OPENAI_API_KEY'], model="gpt-4", text_input_placeholder="How can I help you?", deployment_id="gpt-4", debug=True )
but I get an error stating
and if I remove the deployment_id, I am getting the error stating engine or deployment_id missing
am I missing something??
@nsvbhat Did you install the chatstream version on that PR?
pip install chatstream@git+https://github.com/wch/chatstream.git@azure
@nsvbhat oh I think azure_deployment_id is the argument you want, not deployment_id.
ok i see that in the library its called azure_deployment_id and not deployment_id .. will trying using that n see.
@trangdata I see some references to a proxy in the stack trace, and I know you mentioned something about a proxy earlier. I don't see anything in the Azure OpenAI docs about using a proxy, though. How are you using it?
I'm setting proxy in app.py along with other openai variables, and this has worked for me in different projects too:
openai.api_type = os.getenv("OPENAI_API_TYPE")
openai.api_base = os.getenv("OPENAI_API_BASE")
openai.api_key = os.getenv("OPENAI_API_KEY")
openai.proxy = "proxy-server.bms.com:8080"
openai.api_version = "2023-03-15-preview"
For both of you, have you tried running the example app in https://github.com/wch/chatstream/pull/10#issuecomment-1641451668 ? Note that you will need to set the openai.api_base URL (https://winstontest.openai.azure.com in the example) as well as a deployment ID (my-gpt-35-turbo in the example).
Yes. This is what my server function looks like:
def server(input: Inputs, output: Outputs, session: Session):
chatstream.chat_server(
"mychat",
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-3.5-turbo",
azure_deployment_id="Turbo",
debug=True,
)
>>> import openai
>>> openai.__version__
'0.27.8'
I have the exact same experience with @iainmwallace when I tested for communication with Azure using acreate (running on vscode interpreter using jupyter). But things are fine with create:
response = openai.ChatCompletion.create(
engine="Turbo",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Tell me about yourself"},
],
temperature=0.7
)
response
<OpenAIObject chat.completion id=chatcmpl-7elyKFl8th1wx0tvBT2B1MbgIVnwm at 0x109497cb0> JSON: {
"id": "chatcmpl-7elyKFl8th1wx0tvBT2B1MbgIVnwm",
"object": "chat.completion",
"created": 1689951880,
"model": "gpt-35-turbo",
"choices": [
{
"index": 0,
"finish_reason": "stop",
"message": {
"role": "assistant",
"content": "As an AI assistant, I am designed to provide helpful information and assist with various tasks. I can answer questions, provide recommendations, assist with scheduling, and more. I am constantly learning and updating my knowledge base to better assist users like you. How can I help you today?"
}
}
],
"usage": {
"completion_tokens": 56,
"prompt_tokens": 21,
"total_tokens": 77
}
}
Error with acreate()
InvalidURL Traceback (most recent call last) File ~/miniforge3/envs/intel-search/lib/python3.11/site-packages/openai/api_requestor.py:668, in APIRequestor.arequest_raw(self, method, url, session, params, supplied_headers, files, request_id, request_timeout) 667 try: --> 668 result = await session.request(**request_kwargs) 669 util.log_info( 670 "OpenAI API response", 671 path=abs_url, (...) 674 request_id=result.headers.get("X-Request-Id"), 675 )
File ~/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/client.py:536, in ClientSession._request(self, method, str_or_url, params, data, json, cookies, headers, skip_auto_headers, auth, allow_redirects, max_redirects, compress, chunked, expect100, raise_for_status, read_until_eof, proxy, proxy_auth, timeout, verify_ssl, fingerprint, ssl_context, ssl, proxy_headers, trace_request_ctx, read_bufsize) 535 assert self._connector is not None --> 536 conn = await self._connector.connect( 537 req, traces=traces, timeout=real_timeout 538 ) 539 except asyncio.TimeoutError as exc:
File ~/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/connector.py:540, in BaseConnector.connect(self, req, traces, timeout) 539 try: --> 540 proto = await self._create_connection(req, traces, timeout) 541 if self._closed:
File ~/miniforge3/envs/intel-search/lib/python3.11/site-packages/aiohttp/connector.py:899, in TCPConnector._create_connection(self, req, traces, timeout) ... 683 raise error.Timeout("Request timed out") from e 684 except aiohttp.ClientError as e: --> 685 raise error.APIConnectionError("Error communicating with OpenAI") from e
APIConnectionError: Error communicating with OpenAI Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...
@trangdata I tried using acreate and seems to be working with minor additions as per this link while using acreate..
`import openai from aiohttp import ClientSession
openai.api_type = "azure" openai.api_base = "your endpoint" openai.api_version = "2023-05-15" openai.api_key = "user key" openai.proxy= "//proxy-server.bms.com:8080"
messages = [ {"role":"system","content":"You are a helpful assistant."}, {"role":"user","content":"Tell me about yourself"} ]
async def main(): openai.aiosession.set(ClientSession()) resp = await openai.ChatCompletion.acreate( model="gpt-4", deployment_id="gpt-4", messages=messages, ) print(resp) await openai.aiosession.get().close()
import asyncio asyncio.run(main())`
Output:
but the same thing does not work for chatstream :(
Thank you so much @wch and @nsvbhat for your help. I finally was able to deploy the app. Two things:
- I needed to use the full proxy URL as "http://proxy-server.bms.com:8080" or at least add "//" in front of my usual proxy: "//proxy-server.bms.com:8080"
- The
modelargument must be the actual OpenAI model. So even though Azure uses "gpt-35-turbo", we still have to specifymodel="gpt-3.5-turbo"(with the dot) in our function. This is not an issue if we use another model like "gpt-4".
Here's the entirety of my app:
import os
import openai
from dotenv import load_dotenv
from shiny import App, Inputs, Outputs, Session, ui
import chatstream
load_dotenv()
openai.api_type = os.getenv("OPENAI_API_TYPE")
openai.api_base = os.getenv("OPENAI_API_BASE")
openai.api_key = os.getenv("OPENAI_API_KEY")
openai.proxy = "http://proxy-server.bms.com:8080"
openai.api_version = "2023-03-15-preview"
app_ui = ui.page_fixed(
chatstream.chat_ui("mychat"),
)
def server(input: Inputs, output: Outputs, session: Session):
chatstream.chat_server(
"mychat",
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-3.5-turbo",
azure_deployment_id="Turbo",
# model="gpt-4", # also works with GPT-4 as deployment name
# azure_deployment_id="GPT-4",
debug=True,
)
app = App(app_ui, server)
I think we can merge #10 now @wch.
A final note: @wch's snippet for testing connection was really helpful. Again, this works when my proxy has "//" in front but errors out otherwise.
import openai
openai.api_type = os.getenv("OPENAI_API_TYPE")
openai.api_base = os.getenv("OPENAI_API_BASE")
openai.api_key = os.getenv("OPENAI_API_KEY")
openai.api_version = "2023-03-15-preview"
openai.proxy = "//proxy-server.bms.com:8080"
resp = await openai.ChatCompletion.acreate(
deployment_id="Turbo",
model="gpt-35-turbo",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Tell me about yourself"},
],
stream=True,
)
print(resp)
async for chunk in resp:
print(chunk)