pnp
💡 [Feature]: Add Language Model Tools for SPFx Toolkit to use in GitHub Copilot Agent mode
🎯 Aim of the feature
The aim is to clarify list of LLM Tools SPFx Toolkit could provide as part of the extension. The idea is to allow to manage SharePoint Online from within VS Code when chatting with GitHub Copilot in agent mode.
Currently how it works (already part of current pre-release) is that each LLM Tool wraps one (but may wrap more) CLI for Microsoft 365 command. We could just provide everything that is possible to manage using CLI for SPO but first of all some commands don't make sense to run from LLM and second of all we need to respect limits. Although there is no confirmed number what is the max number of tools we may have in agent mode it is clear that the more tools we have the worse as GitHub Copilot Agent may just get lost and may not know what tool should be used so in order to avoid that we should create tools only for the things we consider that might be helpful for SPFx development (since it's SPFx Toolkit) and each tool should have very good description (short and clear) for the LLM so Copilot will know what to use.
What I was initially thinking was providing a totally new way of provisioning the needed resources for SPFx development. So usually we need to create some sites, lists or libraries, some list items, pages, maybe change some settings, create some content types or maybe manage term store values, add and install apps (webparts) etc. Currently I added a few tools that allow to do basic operations on site, lists, apps and add a page but lets explore if those are actually needed and what tools would be best to have.
Some things to consider
- Lets use CLI for Microsoft 365 SPO commands to check what is possible as most probably each tool will use one or more of those commands.
- Lets set a hard number limit to max 60 tools (for now, maybe we could think of 70-80 tools but no more for sure)
- Lets also think of other tools that might be helpful that are not connected to managing SharePoint, for example to search the PnP SPFx webpart samples repo for a sample most similar to our prompt and show or download it for the user.
- After we confirm the tools lets group them into smaller sub issues and get it shipped
📷 Images (if possible) with expected result
The aim is to extend GitHub Copilot Agent with tools that it may use to resolve user prompt to take action
demo how it works -> https://1drv.ms/v/c/e82bbd5e6a08f219/EXYLrH3aOUlDkKBQbrYibpwBIWsVp_qCRLZULAginYZxFw?e=gJ2R1v
🤔 Additional remarks or comments
follow up for initial work done in issue #441
Initial commit that added this feature https://github.com/pnp/vscode-viva/commit/665355dfd319f3a9c9c5802606fc914b95cc76b7 It is quite straight forward and easy.
docs: https://code.visualstudio.com/api/extension-guides/tools
@Saurabh7019, @nicodecleyre I would really love your input on this 👀🙏 let me know if you need some short call or chat to discuss it internally. If you also want to see it in action join me on the upcoming viva call this week or check the recording day after I plan to show it in action as well 😉
If we are worried about Agent getting confused by having too many tools then 60-70 tools would also be a lot. It will only be clear with testing, but I think it shouldn't be a problem. SPO commands are clearly defined and not ambiguous, using the same structure as you have started should make agent's job easier.
This view, however, could appear a bit cluttered. Is it possible to group the tools here?
If we are worried about Agent getting confused by having too many tools then 60-70 tools would also be a lot. It will only be clear with testing, but I think it shouldn't be a problem. SPO commands are clearly defined and not ambiguous, using the same structure as you have started should make agent's job easier.
This view, however, could appear a bit cluttered. Is it possible to group the tools here?
No, not yet. But there is a lot of user voice and feedback to allow to make group of tools within an extension or MCP server and let LLM decide based on the prompt which group should ba active. But I expect it may take some time as there is a loooot going on the MCP server community and there are bigger problems than this currently to solve
I was thinking of maybe the following tools:
- spo search - to query anything from SPO for additional context
- app - add/get/list/deploy/install/unisntall/remove
- contettype - add/get/list/remove/set and fields
- field - add/get/list/remove/set
- file - list/get/move/copy/remove
- folder - list/get/remove/add
- group - add/get/list/remove + member add/list/remove
- list - add/get/list/remove/set + list contenttypes commands maybe also views?
- listitem - add/get/list/remove/set
- page - add/get/list/remove
- site - add/get/list /remove + appcatalog add/remove/get
the above is already more or less 60 tools 🤔
Maybe also we could consider:
- feature - list/enable/disable
- applicationcustomizer - add/get/list/remove/set
- commandset- add/get/list/remove/set
- term and term group and term set - add/get/list
- maybe we could consider unique permissions
Besides managing SharePoint I was considering some tools that will help in developing SPFx solutions:
- for example search for SPFx sample from PnP repo that is most similar to the one specified in the prompt
- ... need to think on that a bit more 😉
Also I think if it wouldn't make sense to allow to sign in when VS Code is not open in context of SPFx project so that users may login anyway and use the LLM tools to manage SharePoint Online
@Saurabh7019, @nicodecleyre did you had a chance to check the list of tools suggested 👆
Also I think if it wouldn't make sense to allow to sign in when VS Code is not open in context of SPFx project so that users may login anyway and use the LLM tools to manage SharePoint Online
also this comment 👆 I think we could create a separate issue from this to allow to login using SPFx Toolkit and use SPFx toolkit chat partcipant and LLM tools without being in the context of the project. Let me know what do you think and I may spec out this issue as well and I think we may open it up whenever it will be double checked by you
I just got a new idea when catching up .Net Conf latest enhancements in upgrade assistant
So for a very long time now we have the spfx project upgrade and spfx project validate commands which we also expose as actions in SPFx Toolkit. The biggest problem (based on feedback I heard many many times) is: "ye but you know what that is only a report and now I need to do that on my own".
Well that got me thinking now what if we create an LLM Tool that would instruct the GitHub Agent to actually perform the required changes based on the upgrade or validate output and try to gulp bundle the app and if it fails as next step try to figure out the issues to see if it may move a bit forward 🤔.
Of course there will be projects were it will fail and still manual work will be needed, but hey that's why we have 'human in the loop' flow with AI.
@Saurabh7019, @nicodecleyre I am super interested in what you think? I suggest this should be a separate issue as well once we clarify this.
@waldekmastykarz I was wondering, since you are like the father of this upgrade/validate process. What do you think of my idea 👆. Thanks for the feed in advance 🙏🙂
I just got a new idea when catching up .Net Conf latest enhancements in upgrade assistant
So for a very long time now we have the
spfx project upgradeandspfx project validatecommands which we also expose as actions in SPFx Toolkit. The biggest problem (based on feedback I heard many many times) is: "ye but you know what that is only a report and now I need to do that on my own". Well that got me thinking now what if we create an LLM Tool that would instruct the GitHub Agent to actually perform the required changes based on theupgradeorvalidateoutput and try togulp bundlethe app and if it fails as next step try to figure out the issues to see if it may move a bit forward 🤔. Of course there will be projects were it will fail and still manual work will be needed, but hey that's why we have 'human in the loop' flow with AI. @Saurabh7019, @nicodecleyre I am super interested in what you think? I suggest this should be a separate issue as well once we clarify this.@waldekmastykarz I was wondering, since you are like the father of this upgrade/validate process. What do you think of my idea 👆. Thanks for the feed in advance 🙏🙂
Great idea! Imo if it fails we should rollback the changes to before the situation before the user clicked on upgrade or validate
I suggest we take a step back and consider:
- who would benefit from using LLM in this context?
- what problem do they have that using an LLM would solve?
- does this solution actually improve their life?
If I take for example the scenario you shared in the image of creating a site: it's for developers who need dev assets, so the first two answers are clear. But then, when I go to answer the last question, I'm not quite convinced of the benefit. Given the time it takes me to explain exactly what kind of site I need, with what templates and pages in prose, have LLM parse my intent, and confirm it with me, I can create the same thing twice using the UI and without a single chance of error. A rich web UI makes it way more efficient (and faster!) to perform this action than using an LLM.
I'm not saying that the whole idea of using LLM is flawed but we should beware of introducing it for the sake of it and rather look at what problems customers actually have where using an LLM would be highly beneficial.
One such scenario that I could name top of mind is creating PnP provisioning templates. If you're not well-versed in them, and only use them sporadically, you'll lose a lot of time trying to write them properly. So being able to explain in a sentence or two what you need, and then have the agent lookup the docs, build the template for you, and confirm it's valid would be awesome! I bet, that there are other such cases too.
As a rule of thumb, if it takes longer to write the prompt than to do the actual thing, it's a clear sign that we're on the wrong track and should revisit the scenario and whether it's suited for using an LLM.
@waldekmastykarz that's a long response.
I was more thinking if you could provide feedback on that specific comment https://github.com/pnp/vscode-viva/issues/449#issuecomment-2832586303
Since you are like the father of spfx upgrade and doctor commands. Not on the whole issue 😉, but I am glad you did at it is always good to get more perspective and feed 👍🤩
To tackle some of the feed you provided at first I had the same kind of thinking as you have now. I also did not saw benefit it this and in many similar MCPs that show up now, like why would I create an issue over a GitHub MCP if just going to the page itself is quicker and the way I do it now. But then I realized that is like 'my bubble's thinking. When I look even at the way I work and compare it somehow to how new devs now work there are huge differences. For example I still google and my younger deva start with Copilot to chat about a problem, Also when I use some LLM tools I type but I noticed more folks now speak. And trust me saying to GH Copilot to create a site and a list with some columns and add a app takes like 10-15s to talk and the prompt is ready. 1 min after it is executed by GH Copilot. And in the VS Code output you get the CLI command that were executed which is a starting point for future provisioning script. I say it is a lot lot faster then doing 20+click and typing to do the same thing manually.
To sum up, things change, and the dynamics ar really fast at the moment, sometimes hard to anticipate if something will be used or stay or not. That is exactly why this feature is still in prototype and beta release to check if and how it will be used and what tools might be helpful. Maybe those that allow to manage SPO or maybe those that will get PnPjs docs and create proper TS code using PnPjs. 🤔
Let's agree to disagree 🙂
As for SPFx upgrade, I think that's a very good idea. We already offer scripts in the upgrade report. It would be awesome to have the agent:
- Ensure that you've got all code committed, and if you don't use git, create a backup copy
- Extract the upgrade script from the report and run it
- Build the project, run tests if you've got any
- Fix what needs fixing if anything
Thanks @waldekmastykarz for your honest feed. I really appreciate it 👍. @Saurabh7019, @nicodecleyre I think based on this I'll start creating sub issues with features that may improve the way we allow to use AI in SPFx Toolkit