dotnet
AI Chatbot for DNCEng Docs: Scale Azure Chatbot & Improve Data Ingestion
- Explore Scaling Azure AI Studio Chatbot to include:
- Matrix of truth
- Info from Issues
- PR Discussions
- Teams Channel info
- Wiki Docs
-
Investigate autonomous index updates for Azure AI Studio chatbot that triggers when new docs are added
-
Ignore stale docs for Azure AI Studio chatbot data ingestion procedure
To evaluate the value of this effort, I analyzed the First Responder (FR) channel questions from the last month. Here are my findings.
Of a total of 52 questions:
- ✅ 3 questions could be answered today by a bot (with our current documentation), resulting in a total time savings of 12 minutes for this month.
- ❓ 8 questions could be addressed by a bot if we significantly invest in documentation, potentially saving up to an hour for this month.
- ❌ 40 questions couldn’t be handled by a bot.
The reasons behind the inability to address these 40 questions vary:
- Many questions are related to tasks like approving permissions, adding secrets, or creating pipelines, all of which require manual intervention.
- Some questions are formulated in a way that a bot wouldn’t be able to follow. For instance, users often send messages like “My build isn’t working; here’s the link,” providing minimal context. Similarly, images accompanied by queries like “Why am I getting this message?” pose challenges for automation.
- Certain queries are complex and unique, falling outside the scope of routine tasks. These often lead to discussions, decision-making, and advice.
- Questions related to recent event, such as system failures, require real-time updates, which may necessitate other bot features.
In summary, based on my brief analysis, the value of implementing a bot for these tasks appears relatively low (saving just over an hour of developer time per month). However, the effort involved is substantial:
- Bot Development: Creating and maintaining the bot.
- Documentation: Reviewing existing documentation and writing new content.
- Event Feeding: Keeping the bot informed about recent events (e.g., outages).
- Guidance: Assisting users in formulating explicit questions, with uncertain outcomes regarding user adoption.
I'd love to see the data (the questions and the could/could not results). Are they someplace I can see?
Some questions are formulated in a way that a bot wouldn’t be able to follow. For instance, users often send messages like “My build isn’t working; here’s the link,” providing minimal context. Similarly, images accompanied by queries like “Why am I getting this message?” pose challenges for automation.
Interesting find. The tricky part here is that if the users were faced with no options other than asking the bot, they would probably put more effort into their prompts when the bot tells them to be less vague. The users would then maybe look at their build errors, and reading the log is sometimes all that was needed in the first place.
I'd love to see the data (the questions and the could/could not results). Are they someplace I can see?
Sure! this is the whole information: .NET Engineering Service bot