Possible answer to private repository support

[S0AndS0]

• [ ] Handle submodules referring to private repos (ssh?)

If I remember correctly the GITHUB_TOKEN lacks permissions for private repositories.

Not all is lost though, checking the permissions of repo ("Full control of private repositories") when generating a new token at https://github.com/settings/tokens/new, and then overwriting the environment variable within Workflows that require more permissions should to the trick...

uses: domdere/git-submodule-action@master
env:
  GITHUB_TOKEN: ${{ secrets.ALL_REPOS_TOKEN }}

Note, above assumes that https://github.com/<maintainer>/<repository>/settings/secrets has been setup with a secret named ALL_REPOS_TOKEN containing the token generated from prompts followed at the previous link.

... SSH setup should only be required if those utilizing submodules are also using SSH URLs; in my experience GitHub automation doesn't support SSH URLs and their documentation, if I remember correctly, recommends HTTPS instead; even for their own domain.

---

Small aside; I think if line 37 entrypoint were...

git submodule update --init --merge --recursive --remote

... it may eliminate the need for lines 38 through 40, though that may also download more than what's necessary to update the .gitmodules file and directory-links.

[domdere]

Cool thanks, yeh I am aware of the "Full control of private repositories" option, but i was specifically talking about supporting repos who use SSH urls for their submodules, despite githubs recommendation, there are good reasons (unrelated to automation) to use them.

But really its a valid mode supported by git so even something minimal like "its my preference" is something you have to accept as a justification coming from someone else.

git submodule update --init --merge --recursive --remote was considered, but I didn't want to recurse into nested submodules, as that can get pretty ugly. My submodules are usually structured so the nested submodules arent necessary

However like I said above i guess thats going to be a common use case for others so it should at least be available as an option to provide support for it

I'll think about that one,

Thanks for the feedback!

[S0AndS0]

Welcome for sure.

What I was getting at is your Action likely is already able to handle private repositories via HTTPS URLs; setup is all Client and Workflow configured. I wont argue that SSH does allow for some extra fanciness, so provided that an action.yml file looks sorta like...

name: 'git-submodule-action'
description: "Bump git submodules on '/submodules' comment"

inputs:
  deploy_key:
    description: 'Private key authenticated to GitHub'
    required: false

runs:
  using: 'docker'
  image: 'Dockerfile'

branding:
  icon: git-pull-request
  color: blue

... the following Bash code should enable SSH authentication to GitHub...

#!/usr/bin/env bash


if [ -n "${INPUT_DEPLOY_KEY}" ]; then
    ## Adds key from Action Input and sets permissions to read/write only for owner
    mkdir -vp "${HOME}/.ssh/github_auth"
    tee -a "${HOME}/.ssh/github_auth" 1>/dev/null <<<"$(printf '%s\n' "${INPUT_DEPLOY_KEY}")"
    chmod 600 "${HOME}/.ssh/github_auth"


    ## Configures SSH authentication for GitHub
    tee -a "${HOME}/.ssh/config" 1>/dev/null <<'EOF'
Host github.com
   HostName github.com
   User git
   IdentitiesOnly yes
   IdentityFile ~/.ssh/github_auth
EOF


    ## Add GitHub to known hosts if needed
    if [ -z "$(ssh-keygen -f "${HOME}/.ssh/known_hosts" -H -F 'github.com')" ]; then
        tee -a "${HOME}/.ssh/known_hosts" 1>/dev/null <<<"$(ssh-keyscan -H 'github.com')"
    fi
fi

Note, thanks be to @bradland for Gist Examples regarding known_host file modifications.

... then Workflows could look a bit like...

uses: domdere/git-submodule-action@master
with:
  deploy_key: ${{ secrets.DEPLOY_KEY }}

... which may or may not be a good idea depending upon permissions that are given to the deploy key used.

Note, while it's possible to support more than GitHub, via something like...

tee -a "${HOME}/.ssh/config" 1>/dev/null <<EOF
Host ${INPUT_HOSTNAME:-github.com}
   HostName ${INPUT_HOSTNAME:-github.com}
   User ${INPUT_USER:-git}
   IdentitiesOnly yes
   IdentityFile ~/.ssh/git_auth
EOF

... I wouldn't advise it.

---

Yeah recursion likely is not needed in this case, maybe git submodules update --init --merge --remote would be sufficient, but that too might download more than what's really needed.

Considering that directories containing a submodule are pointers/hashes to Git, there's likely some clever way of retrieving the latest commit reference for each submodule instead of downloading the whole history of changes.

Adding --depth=10 (or a configurable depth) to the fetch command could help those that are using thoroughly committed submodules.

[fjmorel]

We've got a number of repos all using another one as a submodule using SSH. I currently checkout the submodule in PR actions using this:

    - uses: actions/checkout@v2

    # Checkout submodule
    - uses: webfactory/[email protected]
      with:
        ssh-private-key: ${{ secrets.GA_CHECKOUT_SUBMODULE }}
    - name: Checkout submodules (with 10-commit history for the next step)
      shell: bash
      run: |
        git submodule sync --recursive
        git submodule update --init --force --recursive --depth=10
      # Verifies if submodule is pointing to a commit on master branch
      # Being equal to origin/master or behind (by up to 10 commits) it is fine
      # Being ahead or totally diverged is not
    - name: Verify pointing to commit on master (will fail if pointing to unmerged commit or one older than 10 commits behind latest)
      shell: bash
      run: |
        cd submodule_folder/
        head=$(git rev-parse HEAD)
        echo "Submodule commit: https://github.com/{org}/{repo}/commit/$head"
        master=$(git rev-parse origin/master)
        echo "Latest master:    https://github.com/{org}/{repo}/commit/$master"
        base=$(git merge-base $head $master)
        echo "Latest in common: https://github.com/{org}/{repo}/commit/$base"
        if [ $head == $master ]
        then exit 0
        elif [ $head == $base ]
        then
          echo "::warning file=.gitmodules,line=1,col=1::Submodule is not pointing to latest commit"
          exit 0
        else
          echo "::error file=.gitmodules,line=1,col=1::Submodule is not pointing to master branch"
          exit 1
        fi

and then do some builds & run unit tests.

I basically force PRs to have the submodule be within the last 10 commits to pass. I would love an easy way to bring them up to latest without manually checking out the branch and making a commit.

[domdere]

Ah, nice, thanks for the suggestion, I'll look into it!

On Sun, 26 Apr 2020, 10:23 am Frederic Morel, [email protected] wrote:

We've got a number of repos all using another one as a submodule using SSH. I currently checkout the submodule in PR actions using this:

- uses: actions/checkout@v2

# Checkout submodule
- uses: webfactory/[email protected]
  with:
    ssh-private-key: ${{ secrets.GA_CHECKOUT_SUBMODULE }}
- name: Checkout submodules (with 10-commit history for the next step)
  shell: bash
  run: |
    git submodule sync --recursive
    git submodule update --init --force --recursive --depth=10
  # Verifies if submodule is pointing to a commit on master branch
  # Being equal to origin/master or behind (by up to 10 commits) it is fine
  # Being ahead or totally diverged is not
- name: Verify pointing to commit on master (will fail if pointing to unmerged commit or one older than 10 commits behind latest)
  shell: bash
  run: |
    cd submodule_folder/
    head=$(git rev-parse HEAD)
    echo "Submodule commit: https://github.com/{org}/{repo}/commit/$head"
    master=$(git rev-parse origin/master)
    echo "Latest master:    https://github.com/{org}/{repo}/commit/$master"
    base=$(git merge-base $head $master)
    echo "Latest in common: https://github.com/{org}/{repo}/commit/$base"
    if [ $head == $master ]
    then exit 0
    elif [ $head == $base ]
    then
      echo "::warning file=.gitmodules,line=1,col=1::Submodule is not pointing to latest commit"
      exit 0
    else
      echo "::error file=.gitmodules,line=1,col=1::Submodule is not pointing to master branch"
      exit 1
    fi

and then do some builds & run unit tests.

I basically force PRs to have the submodule be within the last 10 commits to pass. I would love an easy way to bring them up to latest without manually checking out the branch and making a commit.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/domdere/git-submodule-action/issues/5#issuecomment-619459085, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXRHUXK222FYU7TUAONDILRON5H3ANCNFSM4JEORG3Q .

[S0AndS0]

@fjmorel Thank you for the example commands!

I think it may be possible to avoid changing directories by utilizing the foreach option with Git's submodule command, eg...

git submodule --quiet foreach git rev-parse HEAD

• --quite suppresses notification(s) of where Git is checking such that it'll only output refs
• foreach runs a command for each submodule

... once before update and then again after, at which point the hashes could be compared for changes.

And by adding --remote to the update command it should be possible to remove git submodule sync --recursive line.

Combined those suggestions may look something like...

    - uses: actions/checkout@v2

    # Checkout submodule
    - uses: webfactory/[email protected]
      with:
        ssh-private-key: ${{ secrets.GA_CHECKOUT_SUBMODULE }}
    - name: Checkout submodules (with 10-commit history for the next step)
      shell: bash
      run: |
        old_submodule_hashes=$(git submodule --quiet foreach git rev-parse HEAD)
        git submodule update --init --force --recursive --remote --depth=10
        new_submodule_hashes=$(git submodule --quiet foreach git rev-parse HEAD)
        if [ "${old_submodule_hashes}" == "${new_submodule_hashes} ]
        then
          printf 'No submodule changes/updates found\n'
          exit 0
        fi

[fjmorel]

I ended up writing an action based on this that worked with my submodule:

name: Comment triggers

on:
  issue_comment:
    types: [created]

jobs:
  update:
    name: Update submodule
    runs-on: ubuntu-latest
    steps:
    # Checkout repo and add SSH key for submodule
    - uses: actions/checkout@v2
    - uses: webfactory/[email protected]
      with:
        ssh-private-key: ${{ secrets.GA_CHECKOUT_SUBMODULE }}
    - name: Update submodules
      shell: sh
      run: |
        # Debug output the whole event
        # cat ${{ github.event_path }}

        # Verify it contains "/submodule" and is a new comment on a PR
        CONFIG_ERROR=0
        ((jq -r ".comment.body" "${{ github.event_path }}" | grep -E "/submodule") \
          && jq -r ".issue.pull_request.url" "${{ github.event_path }}" \
          ) || exit ${CONFIG_ERROR}
        if [ "$(jq -r ".action" "${{ github.event_path }}")" != "created" ]; then
          echo "not a new comment"
          exit ${CONFIG_ERROR}
        fi

        # Get PR commit ref
        BASE_URI="https://api.github.com"
        API_HEADER="Accept: application/vnd.github.v3+json"
        AUTH_HEADER="Authorization: token ${{ secrets.GITHUB_TOKEN }}"
        PR_INFO=$(curl -X GET -s -H "${AUTH_HEADER}" -H "${API_HEADER}" "${{ github.event.issue.pull_request.url }}")
        REF=$(echo "${PR_INFO}" | jq -r .head.ref)

        # Checkout branch
        git config --global user.email "[email protected]"
        git config --global user.name "GitHub Submodules Action"
        git fetch --all --depth=1 -p
        git checkout -b ${REF} --track origin/${REF}

        # Update submodule
        git submodule init
        git submodule update
        git submodule foreach 'git fetch --all -p --depth=1'
        git submodule foreach 'git checkout origin/master'

        # Commit and push
        git add -v .
        git commit -m "Update submodules to latest" --allow-empty
        git push origin "${REF}"

        # Leave a note
        BODY='{"body":"Submodule updated to latest commit"}'
        curl -X POST -s -H "Content-Type: application/json" \
          -H "${AUTH_HEADER}" -H "${API_HEADER}" \
          --data "${BODY}" \
           "${{ github.event.issue.comments_url }}"

There's probably a cleaner way to do this, but this works for me.