circleci-cli icon indicating copy to clipboard operation
circleci-cli copied to clipboard

Published 20 hours ago • verified publisher icon CircleCI-Public

[circleci local execute] error looking up cgroup: not implemented for cgroup v2 unified hierarchy - 2022 edition

Open matschaffer opened this issue 3 years ago • 15 comments
trafficstars

Meta:

CircleCI CLI Version: 0.1.16947+263902a (homebrew)

Operating System: macOS Monterey - 12.3 (21E230)

Model Name: MacBook Pro Model Identifier: MacBookPro16,1 Processor Name: 8-Core Intel Core i9 Processor Speed: 2.4 GHz Number of Processors: 1 Total Number of Cores: 8 L2 Cache (per Core): 256 KB L3 Cache: 16 MB Hyper-Threading Technology: Enabled Memory: 64 GB System Firmware Version: 1731.100.130.0.0 (iBridge: 19.16.14242.0.0,0)

CircleCI CLI Diagnostic:

Debugger mode: false
Config found: /Users/matschaffer/.circleci/cli.yml
API host: https://circleci.com
API endpoint: graphql-unstable
OK, got a token.
Trying an introspection query on API...
Ok.
Hello, Mat Schaffer.

Current behavior:

When running from https://github.com/LLK/scratch-gui

❯ circleci local execute
Docker image digest: sha256:ea206e01751a9d4acc6dc21fb8184bdf67b0c2a524c156e526bdb991b1bd2e23
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.13
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.104-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Starting container circleci/node:10-browsers
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image is cached as circleci/node:10-browsers, but refreshing...
10-browsers: Pulling from circleci/node
Digest: sha256:f6e04fb16889bc6c91eda5aeaa62bfa2c995bb1a6f7124db358a148fd05418b5
Status: Image is up to date for circleci/node:10-browsers
circleci/node:10-browsers:
  using image circleci/node@sha256:f6e04fb16889bc6c91eda5aeaa62bfa2c995bb1a6f7124db358a148fd05418b5
  pull stats: Image was already available so the image was not pulled
  time to create container: 62ms
Creating Docker containers in parallel
Time to upload agent and config: 1.6537866s
Error:
Unexpected environment preparation error: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/e5e342f52c6bd355332860b6fd6dab41a3f1592032a87675618a92233cd804fc/0f75c7c4b5ddf81be5546a18347ee417666919a5315ac542f279232b3098a1c8" with domain controllers -- it is in an invalid state: unknown

Step failed
Task failed
Error: Unhandled prepare executor error: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/e5e342f52c6bd355332860b6fd6dab41a3f1592032a87675618a92233cd804fc/0f75c7c4b5ddf81be5546a18347ee417666919a5315ac542f279232b3098a1c8" with domain controllers -- it is in an invalid state: unknown

Expected behavior:

Build should start.

When did this begin / Was this previously working?:

Not previously working, but I only started using the circleci CLI this week.

Additional Information:

❯ docker version
Client:
 Cloud integration: v1.0.22
 Version:           20.10.13
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 10 14:08:44 2022
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.6.0 (75818)
 Engine:
  Version:          20.10.13
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       906f57f
  Built:            Thu Mar 10 14:06:05 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.10
  GitCommit:        2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Workaround

I was able to get the build started by using the sha found in https://github.com/CircleCI-Public/circleci-cli/issues/589#issuecomment-1027097706

You'll need to downgrade the CLI first since support for the file was removed in bf781e7d37263bceacc340e9553046d152605c24

  • Uninstall with brew uninstall circleci
  • Download the old Homebrew formula for v0.1.17087 and install from that file as a local recipe 
    curl -LO https://raw.githubusercontent.com/Homebrew/homebrew-core/abe75ec1a5b53b31f8c88eff4b6f402008e1bf46/Formula/circleci.rb
brew install circleci.rb

❯ cat > ~/.circleci/build_agent_settings.json <<JSON
{"LatestSha256":"sha256:008ba7f4223f1e26c11df9575283491b620074fa96da6961e0dcde47fb757014"}
JSON

❯ circleci local execute
Docker image digest: sha256:008ba7f4223f1e26c11df9575283491b620074fa96da6961e0dcde47fb757014
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.13
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.104-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

Starting container circleci/node:10-browsers
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image is cached as circleci/node:10-browsers, but refreshing...
10-browsers: Pulling from circleci/node
Digest: sha256:f6e04fb16889bc6c91eda5aeaa62bfa2c995bb1a6f7124db358a148fd05418b5
Status: Image is up to date for circleci/node:10-browsers
circleci/node:10-browsers:
  using image circleci/node@sha256:f6e04fb16889bc6c91eda5aeaa62bfa2c995bb1a6f7124db358a148fd05418b5
  pull stats: Image was already available so the image was not pulled
  time to create container: 46ms
Creating Docker containers in parallel
Time to upload agent and config: 1.6189462s
Time to start containers: 337.906ms
====>> Preparing environment variables
Using build environment variables:
  BASH_ENV=/tmp/.bash_env-localbuild-1647572688
  CI=true
  CIRCLECI=true
  CIRCLE_BRANCH=
  CIRCLE_BUILD_NUM=
  CIRCLE_JOB=build
  CIRCLE_NODE_INDEX=0
  CIRCLE_NODE_TOTAL=1
  CIRCLE_REPOSITORY_URL=
  CIRCLE_SHA1=
  CIRCLE_SHELL_ENV=/tmp/.bash_env-localbuild-1647572688
  CIRCLE_WORKING_DIRECTORY=~/repo


The redacted variables listed above will be masked in run step output.====>> Restoring cache
Error:
skipping cache - error checking storage: not supported

Step failed
====>> Checkout code
Making checkout directory "/home/circleci/repo"
Copying files from "/tmp/_circleci_local_build_repo" to "/home/circleci/repo"

So it seems that the latest circleci/picard image has a regression that effects at least docker for mac.

matschaffer avatar Mar 18 '22 03:03 matschaffer

oh yes updating the ~/.circleci/build_agent_settings.json with that specific SHA works.. thanks so much!

Kayden-lolasery avatar Mar 22 '22 09:03 Kayden-lolasery

Meta

Meta

Operating System: macOS Monterey - 12.2.1 (21D62)

Model Name: MacBook Pro Model Identifier: MacBookPro12,1 Processor Name: Dual-Core Intel Core i5 Processor Speed: 2,7 GHz Number of Processors: 1 Total Number of Cores: 2 L2 Cache (per Core): 256 KB L3 Cache: 3 MB Hyper-Threading Technology: Enabled Memory: 8 GB System Firmware Version: 428.60.3.0.0

CircleCI CLI Version: 0.1.17142+0ae0a84 (homebrew)

---
CircleCI CLI Diagnostics
---
Debugger mode: false
Config found: /Users/mason/.circleci/cli.yml
API host: https://circleci.com
API endpoint: graphql-unstable
OK, got a token.
Trying an introspection query on API... 
Ok.
Hello, Mason.

> docker version
Client:
 Cloud integration: v1.0.23
 Version:           20.10.14
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 24 01:49:20 2022
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.7.0 (77141)
 Engine:
  Version:          20.10.14
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       87a90dc
  Built:            Thu Mar 24 01:46:14 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.11
  GitCommit:        3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Issue

It doesn't work for me. CircleCI CLI ignores the ~/.circleci/build_agent_settings.json file.

> cat ~/.circleci/build_agent_settings.json 
{"LatestSha256":"sha256:008ba7f4223f1e26c11df9575283491b620074fa96da6961e0dcde47fb757014"}

Docker image digest diverges.

> circleci local execute
Fetching latest build environment...
Docker image digest: sha256:8160cddd8f228496ebaf542d0a5129a90debbde77219e31d8c94d0ea068ead0f
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.14
 Storage Driver: overlay2
  Backing Filesystem: extfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Kernel Version: 5.10.104-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64

masonlouchart avatar Apr 15 '22 22:04 masonlouchart

Similarly, circleci local execute doesn't work on my Intel 2019 MacBook Pro with v0.1.17183 or v0.1.17142, though v0.1.17087 works.

circleci diagnostic 
---
CircleCI CLI Diagnostics
---
Debugger mode: false
Config found: /Users/ryankienstra/.circleci/cli.yml
API host: https://circleci.com
API endpoint: graphql-unstable
OK, got a token.
Trying an introspection query on API... 
Ok.
Hello, Ryan Kienstra.
Machine 
Model Name:	MacBook Pro
  Model Identifier:	MacBookPro16,1
  Processor Name:	8-Core Intel Core i9
  Processor Speed:	2.4 GHz
  Number of Processors:	1
  Total Number of Cores:	8
  L2 Cache (per Core):	256 KB
  L3 Cache:	16 MB
  Hyper-Threading Technology:	Enabled
  Memory:	64 GB
  System Firmware Version:	1731.100.130.0.0 (iBridge: 19.16.14243.0.0,0)
  OS Loader Version:	540.100.7~23
  Serial Number (system):	C02ZT0R5MD6T
  Hardware UUID:	E3C4EE86-8CD0-5E70-8AD2-C9552DB1B425
  Provisioning UDID:	E3C4EE86-8CD0-5E70-8AD2-C9552DB1B425
  Activation Lock Status:	Enabled

$ circleci local execute --job php-test-7.3

...

Error: 
Unexpected environment preparation error: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: cannot enter cgroupv2 "/sys/fs/cgroup/docker/17a27748567cb754976fca9c8c2366d65bc6235308785641092932024931bf17/e6d40747abd04130414779b311791222067ab1913f7588d90f803c2ced3abb65" with domain controllers -- it is in an invalid state: unknown

Step failed
Task failed

kienstra avatar Apr 22 '22 14:04 kienstra

@masonlouchart @kienstra I ran into the same issue - the workaround here no longer works, because the code that reads build_agent_settings.json was removed in this commit.

wrigby avatar Apr 22 '22 20:04 wrigby

Ah, that makes sense. Thanks!

kienstra avatar Apr 22 '22 20:04 kienstra

For any others finding this, I should clarify my above comment - this workaround does work if you first downgrade circleci-cli.

For me, since I installed with Homebrew, this looked like:

  1. Uninstall with brew uninstall circleci
  2. Download the old Homebrew formula for v0.1.17087 and install from that file as a local recipe
    • curl -LO https://raw.githubusercontent.com/Homebrew/homebrew-core/abe75ec1a5b53b31f8c88eff4b6f402008e1bf46/Formula/circleci.rb
    • brew install circleci.rb
  3. Populate build_agent_settings.json
    • echo '{"LatestSha256":"sha256:008ba7f4223f1e26c11df9575283491b620074fa96da6961e0dcde47fb757014"}' > ~/.circleci/build_agent_settings.json

Hopefully this helps anyone trying to get local testing working today.

wrigby avatar Apr 22 '22 23:04 wrigby

Oh nice, good idea to brew install a previous version of the CLI.

kienstra avatar Apr 23 '22 00:04 kienstra

nice one @wrigby - I copied you're notes into the description as well.

matschaffer avatar Apr 24 '22 13:04 matschaffer

The workaround works for me as well; one suggestion: Use brew pin circleci to prevent the next brew upgrade from breaking it :)

ankon avatar Jun 03 '22 07:06 ankon

If anyone is using Linux, this was the solution:

curl -fLSs https://raw.githubusercontent.com/CircleCI-Public/circleci-cli/master/install.sh | VERSION=0.1.16947 sudo -E bash

src-r-r avatar Jun 25 '22 17:06 src-r-r

I am still having this issue on my Mac with Docker Desktop v4.13.0

dantetwc avatar Oct 26 '22 08:10 dantetwc

The solution as described in the comment above works for me. Thanks! 👍

My environment: Docker Desktop 4.13.0 (89412) MacOS Monterey 12.5.1 (21G83) Processor: 2,3 GHz 8-Core Intel Core i9

fvmsiilo avatar Oct 27 '22 14:10 fvmsiilo

FYI, a recent pr put back the possibility to configure the build-agent version with the ~/.circleci/build_agent_settings.json like the cli used to. So you can unpin you circleci cli version in your brew to use the latest version.

Please note that it is not intended to remains this way, this configuration will surely be removed soon after the problem with build-agent is fixed

JulesFaucherre avatar Jan 24 '23 10:01 JulesFaucherre

The workaround solution worked for me.

It's going to be almost one year since the issue was creative though. Any updates from the circle-ci team?

diogogmt avatar Mar 07 '23 00:03 diogogmt

thanks. worked for me. at first i used a different SHA but then figured my mistake.

just make sure to use this SHA that ends with ...014:

echo '{"LatestSha256":"sha256:008ba7f4223f1e26c11df9575283491b620074fa96da6961e0dcde47fb757014"}' > ~/.circleci/build_agent_settings.json

sanginovs avatar Mar 13 '23 00:03 sanginovs