Unable to check out SSH submodule using deploy key

[daveespo]

Moving this thread from #116 over here

@ericsciple -- just tried the new ssh-key param but no joy .. (the SSH private key that I'm passing in via Secret is one that works with the ssh-agent hack I described earlier in this thread)

2020-03-11T20:08:31.0562996Z Current runner version: '2.165.2'
2020-03-11T20:08:31.0564725Z Prepare workflow directory
2020-03-11T20:08:31.0791957Z Prepare all required actions
2020-03-11T20:08:31.0806683Z Download action repository 'actions/checkout@master'
2020-03-11T20:08:33.0116518Z Download action repository 'sfdx-actions/setup-sfdx@v1'
2020-03-11T20:08:33.2496238Z Download action repository 'nanasess/setup-chromedriver@master'
2020-03-11T20:08:40.5559823Z Download action repository 'actions/upload-artifact@v1'
2020-03-11T20:08:41.1828303Z ##[group]Run actions/checkout@master
2020-03-11T20:08:41.1828907Z with:
2020-03-11T20:08:41.1829353Z   lfs: true
2020-03-11T20:08:41.1831532Z   ssh-key: ***
2020-03-11T20:08:41.1831924Z   submodules: true
2020-03-11T20:08:41.1832300Z   repository: <redacted>
2020-03-11T20:08:41.1832778Z   token: ***
2020-03-11T20:08:41.1833150Z   ssh-strict: true
2020-03-11T20:08:41.1833521Z   persist-credentials: true
2020-03-11T20:08:41.1833910Z   clean: true
2020-03-11T20:08:41.1834281Z   fetch-depth: 1
2020-03-11T20:08:41.1834679Z ##[endgroup]
2020-03-11T20:08:41.6235786Z Added matchers: 'checkout-git'. Problem matchers scan action output for known warning or error strings and report these inline.
2020-03-11T20:08:41.6237082Z Syncing repository: <redacted>/patronsignup
2020-03-11T20:08:41.6237958Z Working directory is '/home/runner/work/patronsignup/patronsignup'
2020-03-11T20:08:41.6310463Z [command]/usr/bin/git version
2020-03-11T20:08:41.6458784Z git version 2.25.1
2020-03-11T20:08:41.6491554Z [command]/usr/bin/git lfs version
2020-03-11T20:08:41.7979462Z git-lfs/2.10.0 (GitHub; linux amd64; go 1.13.4)
2020-03-11T20:08:41.8001760Z Deleting the contents of '/home/runner/work/patronsignup/patronsignup'
2020-03-11T20:08:41.8013376Z [command]/usr/bin/git init /home/runner/work/patronsignup/patronsignup
2020-03-11T20:08:41.8095844Z Initialized empty Git repository in /home/runner/work/patronsignup/patronsignup/.git/
2020-03-11T20:08:41.8104612Z [command]/usr/bin/git remote add origin [email protected]:<redacted>/patronsignup.git
2020-03-11T20:08:41.8155136Z [command]/usr/bin/git config --local gc.auto 0
2020-03-11T20:08:41.8200346Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2020-03-11T20:08:41.8240012Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2020-03-11T20:08:41.8606381Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2020-03-11T20:08:41.8641874Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2020-03-11T20:08:41.8916066Z Temporarily overriding GIT_SSH_COMMAND="/usr/bin/ssh" -i "$RUNNER_TEMP/3e0953a3-c730-442d-9693-4efc5a4164fb" -o StrictHostKeyChecking=yes -o CheckHostIP=no -o "UserKnownHostsFile=$RUNNER_TEMP/3e0953a3-c730-442d-9693-4efc5a4164fb_known_hosts"
2020-03-11T20:08:41.8921024Z [command]/usr/bin/git config --local core.sshCommand "/usr/bin/ssh" -i "$RUNNER_TEMP/3e0953a3-c730-442d-9693-4efc5a4164fb" -o StrictHostKeyChecking=yes -o CheckHostIP=no -o "UserKnownHostsFile=$RUNNER_TEMP/3e0953a3-c730-442d-9693-4efc5a4164fb_known_hosts"
2020-03-11T20:08:41.8960017Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2020-03-11T20:08:41.9006512Z [command]/usr/bin/git lfs install --local
2020-03-11T20:08:41.9487647Z Updated git hooks.
2020-03-11T20:08:41.9488142Z Git LFS initialized.
2020-03-11T20:08:41.9510631Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +c485eed0d330b13b19d081fb186d2f35836986ae:refs/remotes/origin/master
2020-03-11T20:08:42.3528109Z ERROR: Repository not found.
2020-03-11T20:08:42.3535717Z ##[error]fatal: Could not read from remote repository.
2020-03-11T20:08:42.3539415Z 
2020-03-11T20:08:42.3540237Z Please make sure you have the correct access rights
2020-03-11T20:08:42.3540662Z and the repository exists.
2020-03-11T20:08:42.3541669Z The process '/usr/bin/git' failed with exit code 128

And my workflow YAML looks like

    steps:
      - uses: actions/checkout@master
        with:
          lfs: true
          ssh-key: ${{ secrets.SFDX_COMMON_SSH_KEY }}
          submodules: true

And my .gitmodules is

[submodule "sfdx-common"]
	path = sfdx-common
	url = [email protected]:<redacted>/sfdx-common.git

[ericsciple]

Does the ssh-key have access to both repos? Or are you using a deploy key (access to only one repo)?

[daveespo]

The ssh-key that I'm supplying to the action/checkout action is only authorized for the submodule (sfdx-common). I presumed that the repo that the workflow is running in would already be implicitly authenticated. Do I have that part wrong?

[ericsciple]

Oh I see. Currently if you supply an SSH key then it's used to checkout the main repo and submodules.

[ericsciple]

Since one of the features of checkout@v2 is to persist the creds on disk to enable easier scripting authenticated git commands, I think using the provided sshkey for the main repo makes sense (remote url is configured with the ssh url and core.sshCommand is setup for auth).

Looking into multiple keys...

[ericsciple]

It looks like ssh allows multiple -i identity_file args.

What do you think about being able to specify multiple ssh keys. For example:

ssh-key: |
  ${{ secrets.my_main_repo_deploy_key }}
  ${{ secrets.my_other_repo_deploy_key }}

I think if one doesnt work, it will fallback and try the next. To be clear, this currently won't work - would need to update the action to support it.

Would that be a good solution?

[daveespo]

I have to back up a second and ask how the checkout action works today ... Is it the case that whatever implicit authentication scheme that is used provides only read-only access to the repo it's running against? i.e. if I were to switch over to using deploy keys for this, would I lose the ability to commit back to the repo? (this isn't a use case I have today, but have been asked by a few team members if we could commit build artifacts back to the repo or to tag a release .. long story)

Second, by switching over to require SSH, it increases the burden of configuration. I need to set up deploy keys for both repos. Not a huge burden, but it does make things more complex.

And then there's that rule about not being able to use the same private key for more than one repo's deploy-key, right? "Key is already in use" if you try to use the same private/pub key pair as a deploy key? That's a huge drag that we've had to work around by creating a dedicated paid Github user (we're on a paid Teams plan) to act as a service user just so we can bless that user with readonly access to a bunch of repositories.

So, yes, what you propose would work, but I think it's going to frustrate a lot of people.

You may need to support multiple SSH keys anyway if a repo has multiple submodules .. so I can see why you're thinking along those lines.

[ericsciple]

how the checkout action works today ... Is it the case that whatever implicit authentication scheme that is used provides only read-only access to the repo it's running against?

The default token - i.e. ${{ github.token }} - token has read/write permission to the main repo and read permission to public repos.

if I were to switch over to using deploy keys for this, would I lose the ability to commit back to the repo?

No. When creating a deploy key, there is a checkbox to Allow write access.

increases the burden of configuration. I need to set up deploy keys for both repos

Have you considered granting the service account read access to the main repo also?

If in the future you want to commit back, the default auth token is still embedded in the config too (http.https://github.com/.extraheader). So if you switch the URL back to HTTPS, then push will use the embedded token. For example: git remote set-url origin https://github.com/my-org/my-repo.git

[daveespo]

It sounds like the only real option is to use the service account to grant access to both repos .. and yes, we can do this.

But again, it sounds more convoluted than dealing with the submodule checkout myself (with the ssh-agent action)

I would definitely prefer for the repo that the workflow is running in to be implicitly authorized and only have to deal with SSH authorization for submodules. But so far it sounds like I'm the only one that's given this a whirl :-)

[ericsciple]

Proposed here: https://github.com/actions/checkout/pull/190

[daveespo]

Great! Though I looked at the PR and it looks just like copy changes? I don't see code changes

[ericsciple]

It was a proposed change to the spec, but prevented other scenarios. As part of the discussion we figured out separate inputs submodules-ssh-key and submodules-ssh-token would be better.

[martinitus]

I would definitely prefer for the repo that the workflow is running in to be implicitly authorized and only have to deal with SSH authorization for submodules. But so far it sounds like I'm the only one that's given this a whirl :-)

Nope, you are not ;-)

I already use a deploy key to pull in a python dependency from another private repository 'B' into the build of our repository 'A'. For this to work, i set up a private key via a configured secret in 'A' and the respective public key in 'B' and use the following step:

      - name: Setup access via public/private key.
        # Below command requires the FOOBAR_PRIVATE_KEY to be configured via github repository secrets.
        # Also the key's public part must be added to the foobar repository deploy keys.
        # A private/public key pair without password (required in this case) can be generated with ssh-keygen.
        # This part is used for the git/foobar part in requirements.txt
        run: |
          mkdir ~/.ssh
          echo "${{ secrets.FOOBAR_PRIVATE_KEY }}" > ~/.ssh/id_rsa
          chmod 600 ~/.ssh/id_rsa

I am tempted to simply clone the submodule in another custom step until the checkout action officially supports it again :)

[paresy]

It looks like ssh allows multiple -i identity_file args.

What do you think about being able to specify multiple ssh keys. For example:

ssh-key: |
  ${{ secrets.my_main_repo_deploy_key }}
  ${{ secrets.my_other_repo_deploy_key }}

I think if one doesnt work, it will fallback and try the next. To be clear, this currently won't work - would need to update the action to support it.

Would that be a good solution?

I like that idea (if it works technically).

On GitHub the problem is, that we need one key for each submodule. AFAIK the idea in PR https://github.com/actions/checkout/pull/190 will only allow one key for all submodules. But we need multiple if we have multiple private submodules. Therefore +1 for this idea.

Thanks!

[Timic3]

I am having the same problem - I have a private repository within organization, which uses multiple public submodules and one private submodule, therefore one key isn't enough.

It would be nice to support multiple SSH keys - at least for now.

[yinonov]

I'm having a similar issue #271 the only difference is my repo - where the workflow fails - resides in the same organization as the submodule's. why should we even provide keys/tokens? aren't both repo on same access level? they use the same ssh key locally

[michaelmwhite]

I'm experiencing this issue as well

[elhedran]

Just another idea for the same core use case.

- uses: actions/checkout@v2
- uses: actions/checkout-submodule@v2
  with: <configuration specific to submodules, not main checkout>

Seems a couple of community actions do something similar already, would prefer a core one for something this basic though.

[elhedran]

or

- uses: actions/checkout@v2
- uses: actions/checkout@v2
  with:
     submodules: [ submodule, paths ]
     ssh-key: ....

and the second uses detects a git repo in the current directory already and doesn't attempt to get that one again, and detects submodules is an array, and only does a subset of modules instead of all of them.

[ustun]

I was able to checkout the submodules without this action, but manually. Here is what I did:

• Create a deploy key in the submodule repo.
• Define the private key for that deploy key as a secret called SSH_KEY_SUBMODULE in the main repo.
• As part of a GitHub action, create a ~/.ssh/config file with the following format:

      - name: create ssh-config
        run: |
          cat << EOF > ~/.ssh/config
          Host github.com-mysubmodulerepo
          HostName github.com
          User git
          IdentityFile ~/.ssh/mysubmodulekey

• As part of a GitHub action, create a .ssh/mysubmodulekey using the secret.

      - name: Create ssh key for my sub module
        run: |
          mkdir -p ~/.ssh
          cat << EOF > ~/.ssh/mysubmodulekey
          ${{ secrets.SSH_KEY_SUBMODULE }}
          EOF
          chmod 700 ~/.ssh/mysubmodulekey

• Update my .gitsubmodules file in another GH action:

      - name: override .gitmodules
        run: |
          cat << EOF > .gitmodules
          [submodule "mysubmodule"]
          path = thirdparty/mysubmodule
          url = [email protected]:MyUser/mysubmodule

The key thing is that github.com-mysubmodule should match whatever is in the .ssh/config.

• Then run the submodule init commands:

     -run: |
          git submodule sync
          git submodule update --init --recursive

• Finally remove the ssh keys just in case the action container is reused (probably it is not, right?)

   - run : |
       rm ~/.ssh/*

Pretty smooth process (!) :) but hey, it worked! Hope this helps someone else. I'm sure there must be an easier solution.

edit: here is a blog post that does something similar: https://github.com/actions/checkout/issues/116#issuecomment-687652901

[mshamaseen]

You'll need to checkout each of them individually. Not that hard as you may expect:

- name: Checkout
  uses: actions/checkout@v3

- name: Checkout private tools
  uses: actions/checkout@v3
  with:
    repository: my-org/my-private-tools
    ssh-key: ${{ secrets.PRIVATE_SSH_KEY }}
    path: my-tools

[orta]

Just want to pass along my working version which will use the correct refs:

     - name: Main Checkout
        uses: actions/checkout@v3

      - name: Clone Submodule
        run: |
          mkdir -p $HOME/.ssh
          echo '${{ secrets.SSH_SUBMODULE_PRIVATE_KEY }}' > $HOME/.ssh/ssh.key
          chmod 600 $HOME/.ssh/ssh.key
          export GIT_SSH_COMMAND="ssh -i $HOME/.ssh/ssh.key"
          git submodule set-url ${{ env.SUBMODULE_PATH }} [email protected]:${{ env.REPO }}.git
          git submodule update --init --recursive
          git submodule set-url ${{ env.SUBMODULE_PATH }} https://github.com/${{ env.REPO }}.git
          unset GIT_SSH_COMMAND
        env:
          SUBMODULE_PATH: src/shared
          REPO: myapp/shared

Based on this Stack Overflow: https://stackoverflow.com/questions/71400819/pulling-from-a-private-submodule-github-repository-using-a-deploy-key

[cuong-tran]

Just want to pass along my working version which will use the correct refs:

Nice, thank you very much.