Speeding up Python API calls?

[bbence84]

trafficstars

Is there any way to speed up the a sequence of completion API calls to the GPT3 models? I have a use case where the response of the completion API should be returned very quickly, but I need the full reponse, so streaming would not help here. Is the python API maybe reusing the HTTP connection or auth tokens accross multiple API calls? Is there maybe a workaround? I am also fine changing the python package files locally for a while. :slight_smile: Thanks!

[ted-at-openai]

There's no perfect solution. What speed are you getting and what speed do you want?

One option is to fine-tune a weaker model like ada. Weaker models tend to be faster.

[ted-at-openai]

In general, it will be nearly impossible to guarantee ~100 ms or less with an API call, unfortunately.