se-2-challenges
se-2-challenges copied to clipboard
scaffold-eth
Add a link to the alpha version of Ch0 GPT assistant at bottom of Readme
Context
Last couple of weeks I've been researching different approaches to create an assistant to help new SRE users through their Challenges. I've tried:
-
Fine tunning a model with SE-2 docs, Challenge 0 Readme and TG Challenge 0 Q/A. Didn't work at all, probably didn't do it right because results were terrible. Could try again in the future with a different base model, but this system requires to develop also a layer to interact with the users (web, telegram bot)
-
Using a local AI (PrivateGPT) as a solution to privacy concerns, giving it contexts (SE-2 docs, Challenge 0 Readme and TG Challenge 0 Q/A). You don't share all your info and the user prompts and answers with a 3rd party, and there is no limit of context to provide, but I don't see an easy solution to open it to the users nor add it as telegram bot
-
Create a GPT with Chat GPT Worked the best, has no cost to us or to the user, but requires to have Chat GPT Plus in order to use it. Has a familiar interface for Chat GPT users.
-
Create an assistant with Chat GPT. Same results as a GPT, can be integrated as a telegram bot easily with their API, but the cost can be elevated for us, since this alpha version has a lot of context (many doc files), and Open AI charges per tokens used. Could be a future iteration if we see this useful and want to open it to 100% of potential users.
For this GPT-Ch0 Alpha Assistant I'm providing this context:
- All SE-2 doc files, as independant .md files
- Challenge 0 Readme as .md file
- Q/A from Challenge 0 Telegram group, only using questions and answers post SE-2 migration, fetched manually. Ch0-Telegram-QA.txt
Proposal
If we see the AI Assistant interesting/useful, I would follow this approach:
-
Add this GPT Ch0 Alpha Assistant to the Challenge 0 readme with this PR, and try to get some feedback from the next SRE users.
-
After gathering feedback for a few weeks, if it feels useful, we can create the Q/A files for Challenges 1 and 2, and open the GPTs for those Challenges.
-
In the future, if we feel that not many SRE users have Chat GPT Plus, and we feel the assistants are useful for BG, we could setup TG bots with a BG Open AI account and buy some credits for the API.
Great job and research, Pablo! I tried to ask some question and it's working good!
I think that info about GPT should be before Checkpoint 0 or somewhere at the beginning, since at the end of the readme users mostly has no questions.
I'm not sure that a lot of users has GPT plus though, but hope it's enough. Because in general it's a great idea and can save a lot of time!
This is cool. I like scrubbing the telegram for answers and maybe a file where we could just enter questions/answers that we know people will ask. Neat stuff.
I've created another assistant with a different tech stack I'm more happy with => https://scaffold-eth-assistant.streamlit.app/
Still very rusty and dirty, but the good thing is it will let others contribute adding or curating questions/answers for each challenge.
Still very rusty and dirty, but the good thing is it will let others contribute adding or curating questions/answers for each challenge.
Great! As I understand it's working only for challenge 0? In any case, I think we need to try it, collect data and make it better!
Merging this! Let's see the feedback we get in Telegram.
Will try to prepare the Challenge 1 Q/A file this week 🙌