unity-builder
unity-builder copied to clipboard
Unable to import packages from private repo
Bug description
Have two packages hosted on github as private repos.
Getting this error:
An error occurred while resolving packages:
Project has invalid dependencies:
<package_id>: Error when executing git command. Failed to add the ECDSA host key for IP address '140.82.114.4' to the list of known hosts (/root/.ssh/known_hosts).
ERROR: Repository not found.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
(140.82.114.4 is a github.com ip)
How to reproduce
Yaml snippet:
# setup ssh agent to access private package repos in manifest
- uses: webfactory/[email protected]
with:
ssh-private-key: |
${{ secrets.SSH_PRIVATE_KEY }}
${{ secrets.SSH_PRIVATE_KEY_TWO }}
# test git clone (this successfully clones the repos, suggesting ssh-agent is setup properly)
# - name: Git Clone Test
# run: |
# git clone [email protected]:<repo_1>.git
# git clone [email protected]:<repo_2>.git
# cd <repo_folder>
# ls
# Build
- name: Build project
uses: game-ci/unity-builder@v2
env:
UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
UNITY_SERIAL: ${{ secrets.UNITY_SERIAL }}
with:
targetPlatform: iOS
sshAgent: ${{ env.SSH_AUTH_SOCK }}
Format in manifest is: ssh://[email protected]/<company>/<project>.git
Expected behavior
I expect it to successfully import these packages
Additional details
Followed these docs: https://game.ci/docs/github/builder#private-github-repositories And ssh-agent docs: https://github.com/webfactory/ssh-agent
As per ssh-agent recommendations found, I also tried manually cloning the repos (see yaml) -- this worked fine, so I don't think there's an issue with my ssh-agent setup. Seems to break only when importing packages. If it matters, I did notice this log when I manually cloned: Warning: Permanently added the ECDSA host key for IP address '140.82.114.3' to the list of known hosts.
(140.82.114.3 is another github ip). Maybe the problem has to do with it failing to do so in unity-builder container?
Also tried webfactory/[email protected]
(what's currently in Game-CI docs).
Have tried other formats in the manifest (one above seems closest to what Unity prescribes) with no luck. Tried the url format in terminal and it seems to work:
Hi <company>/<project>! You've successfully authenticated, but GitHub does not provide shell access.
So as far as I can tell, I have everything configured correctly on my end. Any ideas?
Based on a similar issue that remains unresolved, I'm attempting to create a workaround in a fork of unity-builder. Created a new step executed by entrypoint.sh.
#
# Add github ips
#
echo ""
echo "###########################"
echo "# Github to known hosts #"
echo "###########################"
echo ""
ssh-keyscan github.com | tee -a /root/.ssh/known_hosts
But it doesn't let me modify a read-only file system:
tee: /root/.ssh/known_hosts: Read-only file system
and ends up with same error. Any way I can make that container file system writable?
I suppose you can try to mount a volume so local folder became /root/.ssh ...
So I managed to get my fork of unity-builder to allow writing to known_hosts (removing instances of :ro
to volume mounting).
Now I think the problem is trying to do more than one private repo. I have a deploy key for each and, as you can see in the yaml, a private ssh key for each.
I still think my ssh-agent configuration is correct. The logs indicate both sets are successfully paired:
Run webfactory/[email protected]
Adding GitHub.com keys to /home/runner/.ssh/known_hosts
Starting ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-ktiwnDCKaAKO/agent.2326
SSH_AGENT_PID=2327
Adding private key(s) to agent
Identity added: (stdin) ([email protected]:<repo_1>.git)
Identity added: (stdin) ([email protected]:<repo_2>.git)
Key(s) added:
256 SHA256:<hash matching what's on github for repo 1> [email protected]:<repo_1>.git (ED25519)
256 SHA256:<hash matching what's on github for repo 2> [email protected]:<repo_2>.git (ED25519)
Configuring deployment key(s)
Added deploy-key mapping: Use identity '/home/runner/.ssh/key-830b92b9919a8837adb227aa2fcbd56efd50b4e2ece0c18a6dffc100a3163f32' for GitHub repository <repo_1>
Added deploy-key mapping: Use identity '/home/runner/.ssh/key-57de830f1c1598eccb3a9a8389b838cf1ac0819d50b4d708c41ffc5d0e0135f2' for GitHub repository <repo_2>
Comment for (public) key '' does not match GitHub URL pattern. Not treating it as a GitHub deploy key.
And if I clone the repos as a github action step outside the unity-builder container, both repos clone successfully and I can see the files are all there.
But within the container, if I have my fork manually clone these repos outside of Unity Package Manager, only the repo tied to the first private ssh key clones successfully. I tested this by swapping which private ssh key was listed first in ssh-private-key
. So it seems that unity-builder's implementation of ssh-agent only expects one pair? Will try to continue looking into it myself but any help would be appreciated.
edit: for context, ssh-agent should be able to handle multiple pairs and I think I've followed their guidelines correctly. As far as I can tell, the logs above confirm it finds the comment in the deploy key to create the guids etc (edit2: I also checked git config and it has the expected insteadof entries they use to route multiple key pairs). But for further clarity, here's how my deploy key is formatted (which seems to fit the formatting standards for comments):
ssh-ed25519 <key> [email protected]:<company>/<project>.git
I have a workaround but it's not ideal -- I just clone the repos outside of the docker container and reference the files locally in the package manifest. Requires me to enable dirty builds, which I'd like to avoid but it's fine for now.
I think this is related to unity-builder not correctly implementing support for multiple keys inside the docker container: https://github.com/webfactory/ssh-agent/issues/78
I tried forking my own unity-builder to apply the suggested fix, got close but couldn't get it working quickly enough so I gave up.
Thank you for posting all your findings.
I'm hoping more people could pitch in here. Over time we should be able to find elegant solutions.
Hello, I am also struggling with the same bug, @justin-pierce could you please show me how exactly you did your workaround? Maybe it will help me out too
Hello, I am also struggling with the same bug, @justin-pierce could you please show me how exactly you did your workaround? Maybe it will help me out too
This clones the package repos locally (you still need to setup ssh-agent correctly for multiple private repos, see my snippets above) and runs a shell script to set the package manifest to point to the local files (and prints the manifest file so you can confirm the changes worked as you intended)
- name: Clone Packages To Use Locally
run: |
git clone [email protected]:<git_url_1>.git
git clone [email protected]:<git_url_2>.git
echo "Editing package manifest"
chmod +x edit_packages.sh
source edit_packages.sh
cat Packages/manifest.json
edit_packages.sh looks like this:
#!/usr/bin/env bash
sudo sed -i "s|<git_url_1>|file:../<package_1>|" Packages/manifest.json
sudo sed -i "s|<git_url_2>|file:../<package_2>|" Packages/manifest.json
the sed
command is simple string replacement -- s|<string to replace>|<string to replace it with>|
Because you're editing stuff after checkout, you'll need to set allowDirtyBuild: true
in the unity-builder step's with
section
you may have to mess with paths depending on your setup -- just make sure the packages are cloned outside the Assets folder and that the package manifest points to the path you cloned them to.
and for clarity, this workaround doesn't require a custom fork of unity-builder
Thank you very much, helped a lot
Why don't you just use the run command, but an extra shell file?
#!/usr/bin/env bash sudo sed -i "s|<git_url_1>|file:../<package_1>|" Packages/manifest.json sudo sed -i "s|<git_url_2>|file:../<package_2>|" Packages/manifest.json
This one works fine for me
- name: Replace package web path with local path in manifest
run: |
unitymanifest='/home/runner/work/UnitySensei/UnitySensei/unity/Packages/manifest.json'
webrepo='https://github.com/vr-bits/UnitySensei.git'
localrepo='file:/home/runner/work/UnitySensei/UnitySensei/package_in_test'
sed -i "s;$webrepo;$localrepo;" $unitymanifest
- name: Display changes in manifest
run: |
cat "/home/runner/work/UnitySensei/UnitySensei/unity/Packages/manifest.json"
Unfortunately, after that I get this error from Unity, although the package.json is exactly in this location:
An error occurred while resolving packages:
Project has invalid dependencies:
de.vrbits.sensei: The file [/home/runner/work/UnitySensei/UnitySensei/package_in_test/package.json] cannot be found
Hello, I am also struggling with the same bug, @justin-pierce could you please show me how exactly you did your workaround? Maybe it will help me out too
This clones the package repos locally (you still need to setup ssh-agent correctly for multiple private repos, see my snippets above) and runs a shell script to set the package manifest to point to the local files (and prints the manifest file so you can confirm the changes worked as you intended)
- name: Clone Packages To Use Locally run: | git clone [email protected]:<git_url_1>.git git clone [email protected]:<git_url_2>.git echo "Editing package manifest" chmod +x edit_packages.sh source edit_packages.sh cat Packages/manifest.json
edit_packages.sh looks like this:
#!/usr/bin/env bash sudo sed -i "s|<git_url_1>|file:../<package_1>|" Packages/manifest.json sudo sed -i "s|<git_url_2>|file:../<package_2>|" Packages/manifest.json
the
sed
command is simple string replacement --s|<string to replace>|<string to replace it with>|
Because you're editing stuff after checkout, you'll need to set
allowDirtyBuild: true
in the unity-builder step'swith
sectionyou may have to mess with paths depending on your setup -- just make sure the packages are cloned outside the Assets folder and that the package manifest points to the path you cloned them to.
and for clarity, this workaround doesn't require a custom fork of unity-builder
works! thanks for this 😎
Those workarounds look all great but I can't believe that this should be the end :-D There needs to be a proper fix for this so we can bypass the verification or inject know hosts from the outside without building custom images.
Anybody found a way to do this? Can I run extra commands from the workflow document before the tests/build are executed?
inject know hosts from the outside without building custom images
What you're suggesting should already be possible between parameters sshAgent
, gitPrivateToken
and the homedir being mounted.
I can't believe that this should be the end :-D There needs to be a proper fix for this
I'm sure there are cases where it doesn't completely work. Both of the parameters were added in a couple of PRs and could probably be improved upon. Contributions aiming to streamline the process are of course very welcome.