Amplify fails to recursively clone git submodules from private repositories

[dancingfrog]

Before opening, please confirm:

• [X] I have checked to see if my question is addressed in the FAQ.
• [X] I have searched for duplicate or closed issues.
• [X] I have read the guide for submitting bug reports.
• [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

App Id

arn:aws:amplify:us-east-1:312512371189:apps/dftk998dzh5qp

Region

us-east-1

Amplify Hosting feature

Backend builds

Describe the bug

Related to #150 , in that I am also using a git submodule that refers to a private repository, EXCEPT in my case the credentials needed to access the submodule repo are the same as those used to access the parent repository (and Amplify has permissions to access both). This should mean that git submodule update is able to run, HOWEVER, there is a bug in the run order of the Amplify "Cloning repository" stage. The runner removes the git credentials immediately after cloning the parent repository and before calling git submodule update as can be verified in the build log:

2022-08-02T16:23:43.388Z [INFO]: Cloning into 'amplify-app'...
2022-08-02T16:23:43.388Z [INFO]: # Switching to commit: ff50e322dcbe6fe5b353069cef010b97bdea9a53
2022-08-02T16:23:43.398Z [INFO]: Note: switching to 'ff50e322dcbe6fe5b353069cef010b97bdea9a53'.
                                 You are in 'detached HEAD' state. You can look around, make experimental
                                 changes and commit them, and you can discard any commits you make in this
                                 state without impacting any branches by switching back to a branch.
                                 If you want to create a new branch to retain commits you create, you may
                                 do so (now or later) by using -c with the switch command. Example:
                                 git switch -c <new-branch-name>
                                 Or undo this operation with:
                                 git switch -
                                 Turn off this advice by setting config variable advice.detachedHead to false
                                 HEAD is now at ff50e32 use ssh for git submodule refs
2022-08-02T16:23:43.452Z [INFO]: Successfully cleaned up Git credentials
2022-08-02T16:23:43.452Z [INFO]: # Checking for Git submodules at: /codebuild/output/src964993672/src/amplify-app/.gitmodules
2022-08-02T16:23:43.453Z [INFO]: # Updating Git submodules...
2022-08-02T16:23:43.814Z [WARNING]: # Unable to update submodules: Error: Command failed: git submodule update
                                    Cloning into '/codebuild/output/src964993672/src/amplify-app/src'...
                                    Host key verification failed.
                                    fatal: Could not read from remote repository.
                                    Please make sure you have the correct access rights
                                    and the repository exists.

Expected behavior

The Amplify CI/CD runner should not clear the git credentials before running git submodule update

Reproduction steps

1. Create a private repository on Github and add a git submodule to it, which references another private Github repository, containing source code for the frontend.
2. Create a new app in amplify and set up CI/CD linked to the parent private Github repo; enable Amplify permissions to access Github repo(s)
3. Commit and allow CI/CD to perform build in the Amplify console.

Build Settings

version: 1
backend:
  phases:
    build:
      commands:
        - '# Execute Amplify CLI with the helper script'
        - amplifyPush --simple
frontend:
  phases:
    preBuild:
      commands:
        - npm ci
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: /dist
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*

Additional information

No response

[rgmann]

I'm running into the same issue. The main project repo and submodule repo are both private and both belong to the same GitHub account. AWS Amplify has been given access to both repositories.

I can provide additional details about my configuration, but it's virtually identical to that described above.

[lixing]

@hloriana same here. it's been working fine in my other apps which setup before. But it failed to Host key verification failed. for the new app. I was thinking to try ssh-keyscan -t rsa github.com >> ~/.ssh/known_hosts, but what's the correct path for the known_hosts?

[carl-wong]

I believe the problem is the script is clearing out the git credentials before grabbing submodules. Here's part of my build log which failed:

2022-09-26T19:09:53.561Z [INFO]: Successfully cleaned up Git credentials
2022-09-26T19:09:53.562Z [INFO]: # Checking for Git submodules at: ***/.gitmodules
2022-09-26T19:09:53.562Z [INFO]: # Updating Git submodules...
2022-09-26T19:09:54.277Z [WARNING]: # Unable to update submodules: Error: Command failed: git submodule update
                                    Cloning into ***...
                                    Host key verification failed.
                                    fatal: Could not read from remote repository.
                                    Please make sure you have the correct access rights
                                    and the repository exists.

[ghost]

Hi @carl-wong @lixing @rgmann 👋🏽 thanks for bringing this issue to our attention. We are investigating this further and determining a workaround for this behavior.

[dunhampa]

I am running into same problem, but in my case the submodule is referencing a public github repo under the github repo I authorized (using OAuth)

Strangely, another public github repo, not from another github user, works as a submodule.

My workaround was to reference the other account public submodule and not the public github under my account.

[akil-rails]

I am running into the same problem, any workarounds or fix for this?

[sanderkranz]

Hi @hloriii I meet the same issue with a private submodule which could be accessed using the same Git credentials. Did you find a workaround?

[arepageek]

Hello @hloriii any updates ? Thanks!

[s01cy]

greetings @hloriii, have you managed to find any solution? thanks

[DeveloperRyou]

Hello! I meet the same problem with private submodules. Is there any solution?

[purevdondov]

Is there any updates on this? We can't access another codecommit repo giving resource permission to the build role.

[Ralf12358]

Any updates?

[purevdondov]

We ended up doing two repos instead of submodules, and creating read only credential for the submodule that clones over https, because cloning over ssh didn't work.

[tech-team-rural-mda]

At the very least we should have the ability to instruct the Cloning repository task for Amplify's build stage to ignore git submodules (i.e., skip git submodule init/update). Is this currently possible using amplify.yml?

It looks like we only have the ability to define phases for frontend, backend during Build (no clone or cloning properties).

[timheilman]

Plus one. It would be great if amplify frontend hosting supported private git submodules when access has been granted to them in github for the amplify fe hosting system. I need to figure a workaround and it's a shame it doesn't just work.

[jarohen]

Apologies for the comment spam, but it doesn't seem like voting has any externally visible effect.

So, any updates? The above proposal to run submodule-update before the creds are cleared seems pretty trivial? At least, if it's not (for security, perhaps), I'd have thought there'd be a message to that effect.

[yatsky]

Same thing here. Looks like Amplify team has ignored this issue with the original member's github account deleted.

[dev-gush]

Ran into this issue. Its so unfortunate that a CI/CD pipeline in 2023 doesn't support submodule cloning.

[chekdotdev]

+1, this is causing a significant issue in our build pipeline.

[NailKhalimov]

+1 Same issue, while cloning repository

[lucasltv]

+1 Same issue here! Any updates??????????

[ncoder]

Still not fixed? My error message looks the same.

[paulnolan7]

+1 Same issue here!

[tfarkas86]

Same issue

[martinvirtel]

I used the suggestion by @lixing (Thanks!) and came up with this solution, which disables git submodules and emulates its functionality by using git clone, cd and git checkout.

version: 1
frontend:
  phases:
    preBuild:
      commands:
        - ssh-keyscan -t rsa github.com >> ~/.ssh/known_hosts
    build:
      commands:
        - git clone https://github.com/mattstratton/castanet.git themes/castanet-v1.22.11 
        - cd themes/castanet-v1.22.11 && git checkout tags/v1.22.11 && cd ../../
        - hugo
  artifacts:
    baseDirectory: public
    files:
      - '**/*'
  cache:
    paths: []

[martinvirtel]

This can only be used for public repos, though. If you want to checkout a private repo, you'd have to add the credentials (e.g. by checking them out from SSM parameter store using aws cli) before cloning etc.

[DominikScholz]

+1, Same issue here

[lucasltv]

Hi guys... i found a solution...

Update the .gitsubmodules file with the https url plus an app token (or app password generated by git provider).

Example:

[submodule "src/@core/common"]
	path = src/@core/common
	url = https://your_user:[email protected]/xpto/lib-common.git