GitHub Repo Q&A Bot sample app is badly broken

[yoDon]

Describe the bug I pointed the GitHub Repo Q&A bot (using text-embedding-ada-002) at a small NextJS TypeScript repo that uses fetch to interact with gpt-3.5-turbo.

When I asked the bot "Does this repo use Python?", instead of responding with some variant of "No", the bot replied

Yes,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,Python,

When I asked the bot "what chat-gpt model does this repo use?", instead of responding with some variant of "it uses gpt-3.5-turbo", the bot responded

1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
1, 1, 1, 1, 1, 1, 1, 1,
, 1,, 1,, 1,, 1,

, 1, 1, 1, 1, 1,, 1,, 1, 1, 1, 1,
, 1, 1, 1, 1, 1,

, 1, 1, 1,, 1,

, 1, 1, 1,,,,,,,,,,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,

All questions I tried asking the app resulted in similarly useless, incorrect, and poorly formed answers.

Expected behavior A clear and concise correct response to the asked question.

Desktop (please complete the following information):

• OS: OSX
• IDE: VSCode

[evchaki]

@hathind-ms, @microsoftShannon can you take a look .

[craigomatic]

I've been trying to repro this without any success so far and need a little more info...

1. Which of Open AI or Azure Open AI you are using?
2. Can you please share a link to the repo you were trying to query?
3. Are you on M1 mac or Intel mac?

[yoDon]

Hi @craigomatic

I'm using OpenAI and running on Apple silicon (M2 MacBook Air running Ventura 13.1)

1. Fresh checkout of https://github.com/microsoft/semantic-kernel.git
2. code .
3. new terminal window
4. cd samples/dotnet/KernelHttpServer/
5. func start --csharp
6. new terminal window
7. cd samples/apps/github-qna-webapp-react/
8. cp .env.example .env
9. yarn install
10. yarn start
11. paste OpenAI key into Web App
12. select text-embedding-ada-002 Completion Model (NOTE: I could find no guidance on what Completion Models were supported, but spotted text-embedding-ada-002 being used in one part of a screen cap video announcement of the sample)
13. click Save
14. Re-enter the OpenAI key and re-select text-embedding-ada-002 this time for the Embedding Model and click Save a second time (NOTE: Again, I could find no guidance on what Embedding Models were supported so I'm reusing the text-embedding-ada-002)
15. enter repo address for a TypeScript NextJS Hello World repo (the first one I found via google search was https://github.com/santoshkaranam/helloworld-nextjs-typescript but presumably any will do) NOTE: Do not include the .git in the repo address (bug? git repos are generally referred to using their .git url not their GitHub url)
16. enter branch name main or master, as appropriate to the repo in question
17. NOTE: UX does not allow entry of Embedding File Types value (bug?)
18. click the Download icon next to the branch name
19. wait for download to complete
20. note that UX still does not allow entry of Embedding File Types value (bug?)
21. Click Next
22. Ask "Does this repo use Python"
23. Receive badly formatted incorrect response

[yoDon]

The steps I posted provide a 100% repro for this issue. I strongly suspect the problem is that devs on the project are using undocumented Completion Model and Embedding Model selections that must be chosen to enable the sample to work.

There is no documentation I can find on what models to use for the example.

There are dozens of choices for each, so >100 combined possibilities to sift through. Assuming I'm correct that model choice is the root of the issue then without guidance there's needlessly bad odds that people not on the Semantic-Kernel team can get this to work.

[craigomatic]

Apologies for the slow reply here, step 12 is your issue text-embedding-ada-002 is only supported as an embedding model, not completion. Try using text-davinci-003 for completion (we are going to switch all samples over to use the chat completion models such as gpt-3.5-turbo soon)

If you're able to confirm this resolves the issue for you we will work on improving the readme. Thank you for giving SK a try :)

[craigomatic]

(we are going to switch all samples over to use the chat completion models such as gpt-3.5-turbo soon)

This has now happened, so please use something like gpt-3.5-turbo if you are working off the latest code in the repository.

[yoDon]

Fresh Checkout.

If I select gpt-3.5-turbo in the first dropdown per your suggestion, I get a red error message and a disabled "Save" button.

If instead I select text-davinci-003 in the first dropdown, as you previously suggested, I get an error popup.

Additionally, my OpenAI API key stats show that this API key was used today. This is the only OpenAI API related app I've used today, so I do think the API key as entered is valid and working because some part of this app was able to successfully hit OpenAI's servers using it.

[alliscode]

Hi @yoDon, this change should fix the issue and has now been merged into main. Let us know if you still see this issue with the latest code.

[yoDon]

Issue status: Not Fixed

@alliscode I'm sorry to say this but while the state of the app is improved I feel pretty strongly that this issue is not resolved.

There are currently 41 "recommended" completion models and 25 "other" completion models. That's 66 possible choices for the first dropdown, of which a casual sampling suggests the vast majority report "This is not a supported completion model. Please select a different one." To the best of my knowledge, the only place where there is even a hint as to which subset of those 66 valid-looking choices might actually work is the conversation stream on this issue. Any dev who fires up this app, pastes in their API key, and selects the first (or second, or third or etc.) "recommended" completion model is going to be presented with an error message and zero guidance on how to resolve the error. Unless that Dev goes searching through the repo issues and finds this issue, their conclusion will almost certainly be "the first official sample app I tried doesn't work - the semantic-kernel project must not work." That is not a success state for an official sample app.

If one does magically choose the (only?) "recommended" completion model that actually works, namely gpt-3.5-turbo, after saving one is then faced with another blank API key field (why would we ever want to use a different API key for completions than embeddings?) and another dropdown with 25 "suggested" and 41 "other" embedding models, again with zero guidance on whether any or all work. Should I use gpt-3.5-turbo here? It's not "suggested" it's merely "other", which doesn't sound promising. As mentioned in the notes above I did manage to find a screen capture video online somewhere where someone used text-embedding-ada-002 with some version of semantic-kernel, so I'm going to try that here (but again, it's not documented anywhere other than here and since it was onscreen only in that video there's no way to search for it).

After making the above guesses about the completion and embedding models, I pointed the app at the same sample repo I linked to in my original repro instructions https://github.com/santoshkaranam/helloworld-nextjs-typescript and asked it the same question I asked in the original repro ("Does this repo use Python?"). This question is, I believe, a reasonable first question to ask and the app responds with

I don't know whether the response is a sign the app is fundamentally still not working or whether it's a sign that more documentation is required to suggest to users what sort of questions they might successfully ask, but in either case this sample app is currently not usable by a developer who is interested in it as a way to learn how to use the semanti-kernel.

It's entirely possible the solution to the remainder of this issue is a documentation commit rather than a code commit, or there may be more code issues remaining. As someone hoping to use semantic-kernel there's no way for me to know.

[alliscode]

We are in the process of simplifying our samples to better demonstrate using the core SK features and as such we are not adding more to this sample. We really appreciate your feedback and contributions, stay tuned for the improved samples :).