Google Gemini Integration

[nimrod4278]

Adding the option to use Gemini as your LLM for gpt-pilot

[senko]

Thanks for the contribution @nimrod4278 :pray:, this is awesome!

Trying GPT Pilot with various LLMs, we've found that other than GPT4, they mostly stumble on our prompts because we're using JSON schemas to define the expected output. How does Gemini fare in that regard?

It would be great if you could post a few screenshot or copy/paste a log showing it working (ie. making it as far as the development steps and then starting to create files/run commants, I'd count that as a success).

[nimrod4278]

Hi Senko,

Thanks for the cool project.

Gemini is doing well regarding JSON schemas. I did implement a retry mechanism for it to try and fix the json if it fails parsing it. I noticed that usually it will get it right when asked to fix the JSON file given.

Attached are some screenshots of it developing a very simple next js app that managed to run.

[image: unnamed.png] [image: unnamed.png][image: unnamed.png][image: unnamed.png][image: unnamed.png]

On Wed, Jan 10, 2024 at 8:06 PM Senko Rašić @.***> wrote:

Thanks for the contribution @nimrod4278 https://github.com/nimrod4278 🙏, this is awesome!

Trying GPT Pilot with various LLMs, we've found that other than GPT4, they mostly stumble on our prompts because we're using JSON schemas to define the expected output. How does Gemini fare in that regard?

It would be great if you could post a few screenshot or copy/paste a log showing it working (ie. making it as far as the development steps and then starting to create files/run commants, I'd count that as a success).

— Reply to this email directly, view it on GitHub https://github.com/Pythagora-io/gpt-pilot/pull/512#issuecomment-1885364102, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHGN5PCPPHCUK2BSPHDWPVDYN3KC7AVCNFSM6AAAAABBUOVRY2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBVGM3DIMJQGI . You are receiving this because you were mentioned.Message ID: @.***>

[DjWarmonger]

@nimrod4278 Screenshots sent by email didn't work.

I am particularly interested in this as we're yet to see Gemini Ultra, which is said to outperform GPT-4. Even if slightly, it it can make or break the app.

[robin-collins]

i believe that an LLM agnostic approach should be considered now that the new king of the hill for code is claude 3, and there are also some amazing opensource coding LLMs available.

[marcuslimdw]

What about a pluggable interface (ModelManager?) that could be something like this?

class ModelManager(metaclass=ABCMeta)

    @classmethod
    def build() -> Self:
        pass

    def generate_completion(data, request_type, project):  # TODO: add types
        pass

There could then be multiple implementations OpenAiModelManager, GeminiModelManager etc. that would each own configuration (loading API keys, etc. from environment variables) and formatting requests to and responses from their APIs, including any necessary retries - I too have experienced Gemini not returning things in the format you originally requested.

[Alignment-Lab-AI]

Thanks for the contribution @nimrod4278 🙏, this is awesome!

Trying GPT Pilot with various LLMs, we've found that other than GPT4, they mostly stumble on our prompts because we're using JSON schemas to define the expected output. How does Gemini fare in that regard?

It would be great if you could post a few screenshot or copy/paste a log showing it working (ie. making it as far as the development steps and then starting to create files/run commants, I'd count that as a success).

hello, significantly delayed response, but i would be interested in training a small open source model for this

[ssnaikr]

@nimrod4278 Have you used gemini 1.5 pro? How were the results?

[ArtikUA]

@nimrod4278

senko in Discord said: "I can't say when the core team might do it (we have a bunch of stuff on our plate) but would love merge this if it gets into mergeable state"

Could you please update it to the state that it can be merged?

[redline187]

@ssnaikr i was using this https://github.com/zuisong/gemini-openai-proxy but i stopped using gemini because it cannot make a simple html login page with google 2fa Candidate was blocked due to RECITATION

[LeonOstrez]

with v0.2 we are handling different LLMs in a new way

[nimrod4278]

with v0.2 we are handling different LLMs in a new way

can you link to it?