Configure copilot setup workflow to use in copilot agent

[AtishayMsft]

Previous Behavior

New Behavior

Preconfiguring Copilot's dev environment before the agent starts by creating a special GitHub Actions workflow file, located at .github/workflows/copilot-setup-steps.yml.

A copilot-setup-steps.yml file looks like a normal GitHub Actions workflow file, but must contain a single copilot-setup-steps job. More details can be found here. https://docs.github.com/en/enterprise-cloud@latest/copilot/customizing-copilot/customizing-the-development-environment-for-copilot-coding-agent#preinstalling-tools-or-dependencies-in-copilots-environment

There is no security risk associated with this workflow as copilot can actually run these steps on its own. But the commands time out 3/5 times and lead to inefficient utilization of LLM cycles and reiterations with human intervention.

A sample PR authored by Copilot is here. #34520

Related Issue(s)

• Fixes #

[github-actions[bot]]

📊 Bundle size report

✅ No changes found

[github-actions[bot]]

Pull request demo site: URL

[Hotell]

There is no security risk associated with this workflow

I wouldn't count on that 100% as LLMs can be unpredictable and start fetching/sharing unintended things.

---

Do we know how it works under the hood? I checked the PR and it seems it tries to install dependencies on every "ask" ?

Will custom worklfow mitigate that ? in sense of creating a "cache" for it ? But then, where is that cache stored ? is that safe ? -> meaning if there would be some exploit in node_modules, is there a way how to manually prune etc ?

While there is no special need of customization as our toolchain is pretty common, I think extending firewall rules might be better option ATM ? https://docs.github.com/en/copilot/customizing-copilot/customizing-or-disabling-the-firewall-for-copilot-coding-agent

WDYT?

[AtishayMsft]

There is no security risk associated with this workflow

I wouldn't count on that 100% as LLMs can be unpredictable and start fetching/sharing unintended things.

Do we know how it works under the hood? I checked the PR and it seems it tries to install dependencies on every "ask" ?

Will custom worklfow mitigate that ? in sense of creating a "cache" for it ? But then, where is that cache stored ? is that safe ? -> meaning if there would be some exploit in node_modules, is there a way how to manually prune etc ?

While there is no special need of customization as our toolchain is pretty common, I think extending firewall rules might be better option ATM ? https://docs.github.com/en/copilot/customizing-copilot/customizing-or-disabling-the-firewall-for-copilot-coding-agent

WDYT?

Copilot agent hallucination is a valid concern applicable to any use case not specific to this one.

Copilot agent is already enabled in the repo. It can already checkout the code and run dev tools.

In this step, we are just preinstalling the dev environment so that the tool can build, run tests, fix errors and repeat. Given that the dev setup takes 10 minutes to complete, the observation is that the output of copilot agent is not usable and needs continuous guidance from the user.

There are some firewall rules configured preventing access to sites like cypress and some chrome tools. Those will need to be looked at and refined regardless of this change.

Moreover, copilot agent can only be triggered by users with write access to the repository. And run in its own ephemeral environment and push changes to branches with names beginning with copilot/ Copilot coding agent does not have access to Actions organization or repository secrets or variables during runtime. Only secrets and variables specifically added to the copilot environment are passed to the agent.

More security practices are outlined in this article. https://docs.github.com/en/enterprise-cloud@latest/copilot/responsible-use-of-github-copilot-features/responsible-use-of-copilot-coding-agent-on-githubcom#security-measures-for-copilot-coding-agent

IMO, we should leverage the tool, continue to monitor and add firewall rules or repo specific measures to improve the security measures.

[Hotell]

yes I agree we should leverage it more, and thanks for additional answers, although I'm still missing answer to these ( if you dont know them no worries):

Do we know how it works under the hood (this information is lacking in official docs - https://docs.github.com/en/copilot/customizing-copilot/customizing-the-development-environment-for-copilot-coding-agent#preinstalling-tools-or-dependencies-in-copilots-environment) ?

• I checked the PR and it seems it tries to install dependencies on every "ask" ?
• Will custom worklfow mitigate that ? in sense of creating a "cache" for it ?
• if it creates the cache, where is that cache stored ?
  • is that safe ? -> meaning if there would be some exploit in node_modules, is there a way how to manually prune etc ?

[AtishayMsft]

yes I agree we should leverage it more, and thanks for additional answers, although I'm still missing answer to these ( if you dont know them no worries):

Do we know how it works under the hood (this information is lacking in official docs - https://docs.github.com/en/copilot/customizing-copilot/customizing-the-development-environment-for-copilot-coding-agent#preinstalling-tools-or-dependencies-in-copilots-environment) ?

• I checked the PR and it seems it tries to install dependencies on every "ask" ?

• Will custom worklfow mitigate that ? in sense of creating a "cache" for it ?

• if it creates the cache, where is that cache stored ?

  • is that safe ? -> meaning if there would be some exploit in node_modules, is there a way how to manually prune etc ?

This is true regardless of this PR. The agent can still install the packages specified in package.json based on user instructions.