OpenAI API against Azure OpenAI Deployment does not work

[jkfnc]

To Reproduce Give a full working code snippet that can be pasted into a notebook cell or python file. Make sure to include the LLM load step so we know which model you are using.

gpt3 = guidance.llms.OpenAI(model="deployment_name":,endpoint=os.environ["AZURE_OPENAI_ENDPOINT"])
print(gpt3)
prompt = guidance('''The best thing about the beach is {{~gen 'best' temperature=0.7 max_tokens=7}}''',llm=gpt3)
prompt = prompt()
prompt

System info (please complete the following information):

• OS : Windows
• Guidance Version (guidance.__version__): guidance 0.0.53

Error: Traceback (most recent call last): File "c:\Users\ABCDEF\NewSource\guidance_test.py", line 10, in gpt3 = guidance.llms.OpenAI(model="deployment_name",endpoint=os.environ["AZURE_OPENAI_ENDPOINT"]) File "C:\Users\ABCDEF\AppData\Roaming\Python\Python310\site-packages\guidance\llms_openai.py", line 113, in init self._tokenizer = tiktoken.get_encoding(tiktoken.encoding_for_model(model).name) File "C:\Users\ABCDEF\AppData\Roaming\Python\Python310\site-packages\tiktoken\model.py", line 71, in encoding_for_model raise KeyError( KeyError: 'Could not automatically map deployment_name to a tokeniser. Please use tiktok.get_encoding to explicitly get the tokeniser you expect.'

Even if we add mapping for cl100k_base in tiktoken, it gives this 404 error.

[somisawa]

@jkfnc AzureOpenAI in guidance/llms/_azure_openai.py uses msal. It seems to need the public enterprise aplication PublicClientApplication, which is one of the Azure service, and seems to be unclear to deal with Azure OpenAI Service in msal.PublicClientApplication in a guidance/llms/_azure_openai.py's manner. Thus, I've temporarily fixed by overwriting guidance/llms/_openai.py as follows.

class OpenAI(LLM):
    cache = LLM._open_cache("_openai.diskcache")
-   def __init__(self, model=None, caching=True, max_retries=5, max_calls_per_min=60, token=None, endpoint=None,
+   def __init__(self, model=None, deployment_id=None, caching=True, max_retries=5, max_calls_per_min=60, token=None, endpoint=None,
                 temperature=0.0, chat_mode="auto", organization=None, allowed_special_tokens={"<|endoftext|>", "<|endofprompt|>"}):
+       self.deployment_id = deployment_id
...
    def _library_call(self, **kwargs):
        """ Call the OpenAI API using the python package.

        Note that is uses the local auth token, and does not rely on the openai one.
        """
        prev_key = openai.api_key
        prev_org = openai.organization
        assert self.token is not None, "You must provide an OpenAI API key to use the OpenAI LLM. Either pass it in the constructor, set the OPENAI_API_KEY environment variable, or create the file ~/.openai_api_key with your key in it."
        openai.api_key = self.token
        openai.organization = self.organization
        
+       if self.deployment_id is not None:
+           openai.api_base = os.environ.get("OPENAI_API_BASE", None)
+           openai.api_type = "azure"
+           openai.api_version = os.environ.get("OPENAI_API_VERSION", None)
            
        if self.chat_mode:
            kwargs['messages'] = prompt_to_messages(kwargs['prompt'])
            del kwargs['prompt']
            del kwargs['echo']
            del kwargs['logprobs']
            # print(kwargs)
-           out = openai.ChatCompletion.create(**kwargs)
+           out = openai.ChatCompletion.create(deployment_id=self.deployment_id, **kwargs)
            out = add_text_to_chat_mode(out)
        else:
-           out = openai.Completion.create(**kwargs)
+           out = openai.Completion.create(deployment_id=self.deployment_id, **kwargs)
        openai.api_key = prev_key
        openai.organization = prev_org
        return out

Then, you can use azure openai like:

gpt4 = guidance.llms.OpenAI(
    model="gpt-4", 
    deployment_id=<your_deployment_id>
)

[ucabvas]

If you don't want to bother touching inside the library, you can also do this monkeypatch:

def library_call_with_engine(**kwargs):
    kwargs['engine'] = "text-davinci-003"
    return guidance.llm._library_call(**kwargs)


# set the default language model used to execute guidance programs
guidance.llm = guidance.llms.OpenAI("text-davinci-003")
guidance.llm.caller = library_call_with_engine

[jackneil]

also note that if you have an OPENAI_API_KEY env var set and in your code type: openai.api_key = os.getenv("AZURE_OPENAI_API_KEY") The guidance.llms.OpenAI() constructor will load the OPEN_API_KEY env var and try to use that. You'll get an invalid authorization error due to that. Make sure NOT to set the openai.api_key in your code ... INSTEAD pass your AZURE_OPENAI_API_KEY to the constructor like this: self.guidance_llm = guidance.llms.OpenAI(self.gptmodel.value, token=os.getenv("AZURE_OPENAI_API_KEY"))

It's due to this code in the OpenAI constructor: ```

fill in default API key value

    if token is None: # get from environment variable
        token = os.environ.get("OPENAI_API_KEY", getattr(openai, "api_key", None))
    if token is not None and not token.startswith("sk-") and os.path.exists(os.path.expanduser(token)): # get from file
        print("Loading token from file.")
        with open(os.path.expanduser(token), 'r') as file:
            token = file.read().replace('\n', '')
    if token is None: # get from default file location
        try:
            with open(os.path.expanduser('~/.openai_api_key'), 'r') as file:
                token = file.read().replace('\n', '')
        except:
            pass

[williambrach]

Hi guys,

Is anyone working on trying to solve this?

[jackneil]

I used the monkey patch method posted here and works fine for now

Jack Neil, MD CEO | Hank AI, Inc. 711 East Main Street, Suite E, Lexington, SC 29072 @.*** (w) +1.904.GET.HANK

(m) +1.803.466.7551

Personal: LinkedIn ( https://www.linkedin.com/in/jackneil/ ) | Twitter ( https://twitter.com/realjackneil ) | Github ( https://github.com/jackneil ) | Dot ( https://dot.cards/jackmd ) | Reddit ( https://www.reddit.com/user/jackusc ) | Discord ( https://discordapp.com/users/318125799456768001 ) HANK: Website ( https://hank.ai ) | LinkedIn ( https://www.linkedin.com/company/hankai/ ) | Twitter ( https://twitter.com/LETHANKHELP ) | Github ( https://github.com/hankai ) | Youtube ( @.*** )

Sent via Superhuman iOS ( @.*** )

On Tue, May 30 2023 at 5:07 AM, William Brach < @.*** > wrote:

Hi guys,

Is anyone working on trying to solve this?

— Reply to this email directly, view it on GitHub ( https://github.com/microsoft/guidance/issues/87#issuecomment-1568064431 ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AGFRM5SEJZNXDZ6CW3FK5IDXIW2GTANCNFSM6AAAAAAYJVM5YM ). You are receiving this because you commented. Message ID: <microsoft/guidance/issues/87/1568064431 @ github. com>

[williambrach]

@jackneil

Could you please share how to do more parameter passing? Like AZURE_OPEN_AI_API_KEY and endpoint url?

I am getting - InvalidRequestError: Resource not found

[jackneil]

I’ll post the code used and quick how to later today when I get back to work computer

[jackneil]

Here ya go. Hopefully this helps Get your endpoint url from your azure service in portal.azure.com Get your openai api key from your azure service in portal.azure.com

Some of these functions were inside a class, thus the 'self' in the functions and some variables. you'll need to edit those

class GPTModel(Enum):
    GPT4 = "gpt-4"
    GPT3_5 = "gpt-3.5-turbo"
    GPT3 = "text-davinci-003"

class GPTModel_Azure(Enum):
    GPT3_5 = "gpt_35_20230527" #"gpt-35-turbo" #USE YOUR DEPLOYMENT ID
    GPT4 = "gpt-4"
    GPT3 = "text-davinci-003"

gptmodel_to_gptmodelazure = {
    GPTModel.GPT4: GPTModel_Azure.GPT4,
    GPTModel.GPT3_5: GPTModel_Azure.GPT3_5
}

#monkeypatch for using azure's openai api endpoints. they expect a deployment_id and engine parameter
    def library_call_with_engine(self, **kwargs):
        kwargs['engine'] = gptmodel_to_gptmodelazure[self.gptmodel].value #have to map the GPTModel (model names openai uses) to the GPTModel_Azure (model name=deploymentid azure openai models use) before getting the text value
        kwargs['deployment_id'] = gptmodel_to_gptmodelazure[self.gptmodel].value
        print("Library call ...")
        return self.guidance_llm._library_call(**kwargs)

 if useAzure:
            openai_api_key = os.getenv("AZURE_OPENAI_API_KEY")
            openai_api_endpoint = os.getenv("AZURE_OPENAI_ENDPOINT")
            self.guidance_llm = guidance.llms.OpenAI(self.gptmodel.value, token=openai_api_key)
            self.guidance_llm.caller = self.library_call_with_engine
            openai.api_type = "azure"
            openai.api_base = openai_api_endpoint #like: https://openai-medicalcoder.openai.azure.com/ or https://hank-openai-europe.openai.azure.com/
            openai.api_version = "2023-05-15" #3-15-preview"
        else:
            if openai_api_key is None: openai_api_key = os.getenv("OPENAI_API_KEY")
            self.guidance_llm = guidance.llms.OpenAI(gptmodel.value) #load up for gpt4, 3, etc
            openai.api_key = openai_api_key
            openai.api_base = "https://api.openai.com/v1"

Then use it something like this:

                    experts = guidance(self.guidance_handlebars_ourformat, llm=self.guidance_llm) 
                    response = experts(doctype=doctype.value)

[slundberg]

Hi all! Sorry this took a while to address. Can you let me know if the following example works when you use 0.0.61? Thanks!

import guidance

llm = guidance.llms.OpenAI(
    'text-davinci-003',
    api_type='azure',
    api_key='sk-...',
    api_base='https://example-endpoint.openai.azure.com',
    api_version='2023-05-15',
    deployment_id='...',
    caching=False
)

program = guidance("""My favorite flavor is{{gen 'flavor' max_tokens=10 stop="."}}""", llm=llm)
program()

[williambrach]

@slundberg thanks for fix!

it works.

deployment_id == model deployment name at https://oai.azure.com/

[Shubham-Sahay]

I am still not able to get the deployment id, can anyone help?