Failure to execute with submodules and checkout

[bpiwowar]

• [x] I have read Contribution Guidelines.
• [x] I have checked for similar issues and haven't found anything relevant.
• [x] This is not a security issue (which should be reported here: https://circleci.com/security/)

Do you want to request a feature or report a bug? Bug

What is the current behavior?

Running circleci local execute with git and submodules, the checkout command does not the same thing as when the test is executed on circleci.com: it does not include the .git repository, and hence the commands used to update the submodule (git submodule update --init --recursive) does not work.

What is the expected behavior? To include the .git repository, or to copy the files from submodules using git ls-files -z --recurse-submodules rather than git ls-files -z

Which version of the CLI and OS are you using? Did this work in previous versions?

latest - did not check previous versions

Please provide the output of circleci version: 0.1.5389+4379bab and circleci diagnostic:

CircleCI CLI Diagnostics
---
Debugger mode: false
Config found: /home/bpiwowar/.circleci/cli.yml
API host: https://circleci.com
API endpoint: graphql-unstable
OK, got a token.
Trying an introspection query on API... 
Ok.
Hello, Benjamin Piwowarski.

If you have any questions, feel free to ping us at @CircleCI-Public/dx-clients.

[eric-hu]

Hi Benjamin, I tried to reproduce this with a plain git repo first (no submodules), and I could see the .git repository.

In my config.yml

version: 2.0
jobs:
  build:
    docker:
      - image: circleci/ruby:2.4-node
    steps:
      - checkout
      - run: ls -la .git

Output:

...

====>> Checkout code
  #!/bin/sh
mkdir -p /home/circleci/project && cp -r /tmp/_circleci_local_build_repo/. /home/circleci/project
====>> ls -la .git
  #!/bin/bash -eo pipefail
ls -la .git
total 56
drwxr-xr-x   7 circleci circleci 4096 Feb 26 06:40 .
drwxr-xr-x   4 circleci circleci 4096 Feb 26 06:40 ..
-rw-r--r--   1 circleci circleci 1017 Feb 26 06:40 COMMIT_EDITMSG
-rw-r--r--   1 circleci circleci   89 Feb 26 06:40 FETCH_HEAD
-rw-r--r--   1 circleci circleci   35 Feb 26 06:40 HEAD
-rw-r--r--   1 circleci circleci   41 Feb 26 06:40 ORIG_HEAD
-rw-r--r--   1 circleci circleci 1387 Feb 26 06:40 config
-rw-r--r--   1 circleci circleci   73 Feb 26 06:40 description
drwxr-xr-x   2 circleci circleci 4096 Feb 26 06:40 hooks
-rw-r--r--   1 circleci circleci  187 Feb 26 06:40 index
drwxr-xr-x   2 circleci circleci 4096 Feb 26 06:40 info
drwxr-xr-x   3 circleci circleci 4096 Feb 26 06:40 logs
drwxr-xr-x 122 circleci circleci 4096 Feb 26 06:40 objects
drwxr-xr-x   5 circleci circleci 4096 Feb 26 06:40 refs
Success!

Did you include the checkout step in your config?

[bpiwowar]

I just tried with a fresh clone

$ git clone https://github.com/experimaestro/experimaestro.git
Cloning into 'experimaestro'...
remote: Enumerating objects: 136, done.
remote: Counting objects: 100% (136/136), done.
remote: Compressing objects: 100% (99/99), done.
remote: Total 56174 (delta 47), reused 85 (delta 31), pack-reused 56038
Receiving objects: 100% (56174/56174), 26.25 MiB | 7.45 MiB/s, done.
Resolving deltas: 100% (35614/35614), done.
$ # modified the .circleci/config.yml (to add a step - run: ls -la .git just after checkout)
$ cd experimaestro
$ circleci  local execute 

Docker image digest: sha256:5eb8cb710920f567403cc4cbd2de16d5ab8fc9a1b4c6357d2e7137fa5f1a4f62
====>> Spin up Environment
Build-agent version 1.0.7981-23afa854 (2019-02-20T13:18:33+0000)
Starting container bpiwowar/experimaestro-test-main
  using image bpiwowar/experimaestro-test-main@sha256:e610e4954d17f67e46df0770ccf489492a24093376ca89f8137ce865df22a621
Starting container bpiwowar/experimaestro-test-remote
  using image bpiwowar/experimaestro-test-remote@sha256:62c8666523252b592da93dcea2845ce0511c61045e45e1b48df03421d3527b9c
====>> Container bpiwowar/experimaestro-test-remote
Service containers logs streaming is disabled for local builds
You can manually monitor container e1de247b5661d9a220672579f80ec3094125cb2960dbbb86baafcfa4b745c94a
====>> Checkout code
  #!/bin/bash -eo pipefail
mkdir -p /root/project && cd /tmp/_circleci_local_build_repo && git ls-files -z | xargs -0 tar -c | tar -x -C /root/project
====>> ls -la .git
  #!/bin/bash -eo pipefail
ls -la .git
ls: cannot access '.git': No such file or directory
Error: Exited with code 2
Step failed
Error: runner failed
{"Runner":true,"level":"error","msg":"runner failed","task-id":"localbuild-1551173725","time":"2019-02-26T09:35:30Z"}
Task failed

Error: task failed

[dnephin]

@eric-hu I think you will have to run circleci update build-agent to reproduce. It is related to #151

I think we'll need to add .git to the list of files we add to the tar archive.

[eric-hu]

@bpiwowar thank you for trying again and thanks @dnephin for the pointer. After running the build-agent update, I can reproduce on my system as well with the same config as my post above:

====>> Checkout code
  #!/bin/bash -eo pipefail
mkdir -p /home/circleci/project && cd /tmp/_circleci_local_build_repo && git ls-files -z | xargs -0 tar -c | tar -x -C /home/circleci/project
====>> ls -la .git
  #!/bin/bash -eo pipefail
ls -la .git
ls: cannot access '.git': No such file or directory
Error: Exited with code 2
Step failed
Error: runner failed
{"Runner":true,"level":"error","msg":"runner failed","task-id":"localbuild-1551342079","time":"2019-02-28T08:21:22Z"}
Task failed
Error: task failed

I'll create a ticket in our internal tracker to capture this work, as it sounds like it needs to happen in another repo.

[jleni]

any updates on this?

[marcomorain]

Logged as CIRCLE-25061

[corinnesollows]

Hello @bpiwowar We are looking to close this issue, but first wanted to reach out to you and verify that this has been addressed.

[abdelDriowya]

closing because no answer from OP