circleci-cli

circleci-cli copied to clipboard

• [ Y ] I have read Contribution Guidelines.
• [ Y ] I have checked for similar issues and haven't found anything relevant.
• [ Y ] This is not a security issue (which should be reported here: https://circleci.com/security/)

Do you want to request a feature or report a bug?

According to the report: https://github.com/CircleCI-Public/circleci-cli/issues/196 it's a bug.

What is the current behavior?

When I run command circleci tests glob "src/test/**/*.java" locally I get an error: Error: Please ensure that circleci-agent is installed, expected this to be called inside a job: exec: "circleci-agent": executable file not found in $PATH

What is the expected behavior?

I should see the output of the split (as it's stated here https://github.com/CircleCI-Public/circleci-cli/issues/196#issuecomment-438050504 ).

Which version of the CLI and OS are you using? Did this work in previous versions?

❯ circleci version
0.1.6949+d951526
❯ circleci diagnostic

---
CircleCI CLI Diagnostics
---
Debugger mode: false
Config found: /Users/szymonrichert/.circleci/cli.yml
API host: https://circleci.com
API endpoint: graphql-unstable
Error: please set a token with 'circleci setup'
You can create a new personal API token here:
https://circleci.com/account/api

Looks like it was fairly intentional as they have a commit for delegating all the test command to circleci-agent

a839b278f3706f3d31dfbdef60235b2bb515ca38

Hey, any update on this?

Is there a way to test splitting locally?

A new version just released, and this is still a problem.

I've been getting around this by defining circleci tests commands from within a job (e.g. in .circleci/config.yml), and running them with circleci local execute --job=my_cli_test_job

This would be great for debugging test splits, rather than needing to ssh into a job directly.

any progress on this?

Reading your documentation and watching the youtube video here, you clearly show this command being run locally to test the globbing pattern.

I see this help output from the command

circleci local --help                                                                                                                                                                                  
Debug jobs on the local machine


Usage:
  circleci local [command]

Available Commands:
  execute     Run a job in a container on the local machine

Flags:
  -h, --help   help for local

Global Flags:
      --host string         URL to your CircleCI host, also CIRCLECI_CLI_HOST (default "https://circleci.com")
      --skip-update-check   Skip the check for updates check run before every command.
      --token string        your token for using CircleCI, also CIRCLECI_CLI_TOKEN
Use "circleci local [command] --help" for more information about a command.

So there is only one command, which is execute... So this doesn't seem to help either.

So, this is either an issue with the command line no longer working for local testing, or the documentation needs to be updated.

Having the same issue here 🤚🏻

Chiming in as I'm also wanting to test out splitting locally.

If it's of any use I think I got an approximate via:

# find the functional tests
find $(pwd) -wholename '**/functional_test*.py'

# for the equivalent
circleci tests glob "**/functional_test*.py

Although it would be nice, if anyone were to find how to install the circleci-agent.

Another thing I added to our config.yml was the following:

echo RUNNING THE FOLLOWING FUNCTIONAL TESTS: `circleci tests glob "**/functional_test*.py"`

example command in config.yml:

          command: |
            . .env/bin/activate
            mkdir /tmp/screenshots
            echo RUNNING THE FOLLOWING FUNCTIONAL TESTS: `circleci tests glob "**/functional_test*.py"`
            CIRCLE_TEST_REPORTS=~ TEST_SCREENSHOT_DIR=/tmp/screenshots python manage.py test --parallel=4 --keepdb `circleci tests glob "**/functional_test*.py" | circleci tests split --split-by=filesize

So at the very least i could see what tests were being globbed.

Hello all,

the circleci tests command is currently a part of CircleCI's "Build Agent", a specific binary that is injected into jobs on CircleCI which provides a modified CircleCI CLI with some commands not available to the local CLI.

circleci tests split in particular requires information that would only exist on CircleCI in order to operate and could not be run locally.

Using the find command as @allen-munsch has, should be a suitable alternative for local globbing. I would be curious to know more about your local debugging use-case which may require splitting.

Using the find command as @allen-munsch has, should be a suitable alternative for local globbing. I would be curious to know more about your local debugging use-case which may require splitting.

can anyone help me doing the find way with tests running with a npm test command?

this is my config file, the tests run on npm test inside the run step of the build, the objective is to split the test files inside the cypress/integration folder.

Using the find command as @allen-munsch has, should be a suitable alternative for local globbing. I would be curious to know more about your local debugging use-case which may require splitting.

can anyone help me doing the find way with tests running with a npm test command?

this is my config file, the tests run on npm test inside the run step of the build, the objective is to split the test files inside the cypress/integration folder.

npm test is not a command itself, but an alias to a different tool. Each tool will (or wont at all) handle test splitting differently.

The glob command from the CLI only generates a list of files, your test runner must be able to accept a list of files to execute. The split command, takes that list (from the glob) and divides it amongst multiple containers. Each container in a parellel run, runs the exact same job but with different values in the glob list.

In your example, your npm test is actually calling the Cypress CLI. Cypress appears to support splitting: https://docs.cypress.io/guides/guides/parallelization.html#Grouping-test-runs (unfortunately this doc page refers to CircleCI 1.0, we will reach out to cypress for an update).

Cypress will automatically balance your spec files across the available machines in your CI provider. Cypress calculates which spec file to run based on the data collected from previous runs. This ensures that your spec files run as fast as possible, with no need for manual configuration.

https://docs.cypress.io/guides/guides/parallelization.html#Balance-strategy It seems here that Cypress might handle the test splitting automatically.

Using the find command as @allen-munsch has, should be a suitable alternative for local globbing. I would be curious to know more about your local debugging use-case which may require splitting.

can anyone help me doing the find way with tests running with a npm test command? this is my config file, the tests run on npm test inside the run step of the build, the objective is to split the test files inside the cypress/integration folder.

npm test is not a command itself, but an alias to a different tool. Each tool will (or wont at all) handle test splitting differently.

The glob command from the CLI only generates a list of files, your test runner must be able to accept a list of files to execute. The split command, takes that list (from the glob) and divides it amongst multiple containers. Each container in a parellel run, runs the exact same job but with different values in the glob list.

In your example, your npm test is actually calling the Cypress CLI. Cypress appears to support splitting: https://docs.cypress.io/guides/guides/parallelization.html#Grouping-test-runs (unfortunately this doc page refers to CircleCI 1.0, we will reach out to cypress for an update).

Cypress will automatically balance your spec files across the available machines in your CI provider. Cypress calculates which spec file to run based on the data collected from previous runs. This ensures that your spec files run as fast as possible, with no need for manual configuration.

https://docs.cypress.io/guides/guides/parallelization.html#Balance-strategy It seems here that Cypress might handle the test splitting automatically.

@KyleTryon in my case it was mostly to determine why a test wasn't running ( later found out that I typo'd the file name ), but was easy to catch once i learned how to use find in a similar way. As far as splitting by filename, i'm guessing there might be something similar via some combo of ls --sort=size, sort, split, or something like that.

i'm guessing there might be something similar via some combo of ls --sort=size, sort, split, or something like that.

You can split by:

• Test Timing Data
• Name
• File size

Or, rather than using the split command, you can also effectively create your own custom mechanic (if needed/desired) as environment variables CIRCLE_NODE_INDEX and CIRCLE_NODE_TOTAL are injected into each job and could be utilized by any custom script. https://circleci.com/docs/2.0/env-vars/#built-in-environment-variables

Why does the entire page on installing it locally even exist?

The primary thing we'd want to use locally doesn't work. Its like the documentation is taunting us - "your container just ain't good enough, sucker".

circleci tests split in particular requires information that would only exist on CircleCI in order to operate and could not be run locally.

@KyleTryon Can you explain why this is, specifically? What is this critical information that a glob needs here. I can't see a good reason for this, but I can think of causes.

The docs for using tests split say that you can pass a job number (for non-timing based splits), so my use case is covered there. It just sounds like it wasn't designed with offline use/testing in mind, and the code base makes it difficult to fix, so we've been waiting 4+ years for this (yes, old issues). Circle has always been slow slow slow (but not travis slow).

circleci local execute <job_name> ie circleci local execute build