Circleci build fails with non-descriptive error message "Error: runner failed (exited with 101)"

[devurandom]

trafficstars

• [x] I have read Contribution Guidelines.
• [x] I have checked for similar issues and haven't found anything relevant.
• [x] This is not a security issue (which should be reported here: https://circleci.com/security/)

Do you want to request a feature or report a bug?

Report a bug

What is the current behavior?

$ cat .circleci/config.yml
version: 2

jobs:
  build:
    docker:
      - image: docker.io/circleci/python:2.7.16
    steps:
      - checkout
      - run: sudo chown -R circleci:circleci /usr/local/bin
      - run: sudo chown -R circleci:circleci /usr/local/lib/python2.7/site-packages
      - restore_cache:
          key: deps9-{{ .Branch }}-{{ checksum "Pipfile.lock" }}
      - run:
          command: |
            sudo pip install pipenv
            pipenv install
      - save_cache:
          key: deps9-{{ .Branch }}-{{ checksum "Pipfile.lock" }}
          paths:
            - ".venv"
            - "/usr/local/bin"
            - "/usr/local/lib/python2.7/site-packages"
      - run:
          command: |
            pipenv run python setup.py test
      - store_test_results:
          path: test-results
      - store_artifacts:
          path: test-results
          destination: tr1

$ circleci config validate
Config file at .circleci/config.yml is valid.

$ circleci build
Docker image digest: sha256:68c7dc342166aad7a45bd9d76ca2dd9015a3161986cf5bf2693fb250b4d11fba
====>> Spin up Environment
Build-agent version 1.0.11151-deadd97a (2019-05-09T22:12:43+0000)
Docker Engine Version: 18.09.6
Kernel Version: Linux 2ca8e750f14c 5.1.1-gentoo #2 SMP Sun May 12 17:41:57 CEST 2019 x86_64 Linux
Starting container docker.io/circleci/python:2.7.16
  using image circleci/python@sha256:66b7a5a542a2f395cc692a01b5b4af229592ea5e65fd40d6d234ee282a3928a3

Using build environment variables
  BASH_ENV=/tmp/.bash_env-localbuild-1557820371
  CI=true
  CIRCLECI=true
  CIRCLE_BRANCH=change/pytest-hypothesis
  CIRCLE_BUILD_NUM=
  CIRCLE_JOB=build
  CIRCLE_NODE_INDEX=0
  CIRCLE_NODE_TOTAL=1
  [email protected]:devurandom/tripl.git
  CIRCLE_SHA1=0128e57ad836f71746ceab604cfb23d427053700
  CIRCLE_SHELL_ENV=/tmp/.bash_env-localbuild-1557820371
  CIRCLE_WORKING_DIRECTORY=~/project

Error: runner failed (exited with 101)
Task failed
Error: task failed

There is no further error message explaining which task failed or what problem occurred or why. Same happens for circleci local execute.

What is the expected behavior?

Either no error should be reported and the build should succeed, or a more descriptive error message should be reported.

Which version of the CLI and OS are you using? Did this work in previous versions?

$ circleci version
0.1.5607+f705856
$ circleci diagnostic

---
CircleCI CLI Diagnostics
---
Debugger mode: false
Config found: ~/.circleci/cli.yml
API host: https://circleci.com
API endpoint: graphql-unstable
OK, got a token.
Trying an introspection query on API... 
Ok.

This is the first time I try it.

[jaqx0r]

Does the local CLI emit any logs anywhere? It would be really nice to be able to see what it's trying to do so we can try to debug this error 101 ourselves. At the moment I can't tell if I'm doing anything wrong, or I have a broken environment, or this is a bug in the CLI.

[marcomorain]

This error message is produced by the build-agent (an internal component at CircleCI), so the fix will have to be made there. cc @glenjamin I'll raise it with the team that own that tool.

[jrahme-cci]

I notice this is an older version of the CLI which in turn will also be using an older version of build-agent, could you please run circleci update, retry the local build and post the results here?

[devurandom]

"does not work" at all :(

❯ circleci update
You are running 0.1.5607
A new release is available (0.1.6949)
Updated to 0.1.6949

❯ circleci config validate
Config file at .circleci/config.yml is valid.

❯ circleci build
Docker image digest: sha256:68c7dc342166aad7a45bd9d76ca2dd9015a3161986cf5bf2693fb250b4d11fba
docker: Error response from daemon: mkdir /var/lib/docker/btrfs/subvolumes/b572570afafc3c342fd9639afb6c7bd707c83e22b0e39b13ccfd495cd8d82879/$HOME: operation not permitted.
See 'docker run --help'.

# Where $HOME is my home directory

❯ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
1b930d010525: Pull complete 
Digest: sha256:f9dfddf63636d84ef479d645ab5885156ae030f611a56f3a7ac7f2fdd86d7e4e
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
[...]
❯ docker run -v $HOME:$HOME debian test -d $HOME ; and echo yes
yes
❯ docker run -v $HOME:$HOME debian test -d x$HOME ; and echo yes
[no output, as expected]

[corinnesollows]

@devurandom is this still occurring?

[devurandom]

I run into a different error now:

❯ podman version
Client:       Podman Engine
Version:      4.1.1
API Version:  4.1.1
Go Version:   go1.18.3
Git Commit:   f73d8f8875c2be7cd2049094c29aff90b1150241
Built:        Mon Jun 20 00:52:03 2022
OS/Arch:      linux/amd64
❯ circleci version
0.1.20323+990c791 (release)
❯ circleci config validate
Config file at .circleci/config.yml is valid.
❯ circleci build
Fetching latest build environment...
Docker image digest: sha256:a16cd1d9b97708cba6ccb2d073966078206939a36f84c7132c7cc3c7a3486d46
✔ docker.io/circleci/picard@sha256:a16cd1d9b97708cba6ccb2d073966078206939a36f84c7132c7cc3c7a3486d46
Trying to pull docker.io/circleci/picard@sha256:a16cd1d9b97708cba6ccb2d073966078206939a36f84c7132c7cc3c7a3486d46...
Error: initializing source docker://circleci/picard@sha256:a16cd1d9b97708cba6ccb2d073966078206939a36f84c7132c7cc3c7a3486d46: reading manifest sha256:a16cd1d9b97708cba6ccb2d073966078206939a36f84c7132c7cc3c7a3486d46 in docker.io/circleci/picard: received unexpected HTTP status: 500 Internal Server Error

Note that I am not using Docker anymore but switched to Podman.