incubator-devlake [Bug][CircleCI] Unable to detect any circleci projects

Search before asking

[x] I had searched in the issues and found no similar issues.

What happened

No projects show when add the CircleCI connection to my devlake project:

I am only able to connect circleci via the webhook or doing db inserts for my actual projects which also leads to issues with no deployment commits showing in cicd_deployment_commits The webhook doesn't give us historical data so I'd love it if the circleci connection could work for our projects. The circleci token is a personal access token with no scopes needing to be specific to give it full read/write access and connection tested successfully

What do you expect to happen

I want my org CircleCI projects to show in the options when i create the CircleCI connection

How to reproduce

Described above and screenshot showing where

Anything else

No response

Version

Latest / all latest beta versions

Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[x] I agree to follow this project's Code of Conduct

Nov 05 '25 13:11 jackwills-op

Hi @jackwills-op! I'm Dosu and I’m helping the incubator-devlake team.

Based on the current implementation, DevLake's CircleCI plugin fetches project data using the CircleCI API, but it relies on a specific ProjectSlug parameter for each project and does not automatically list all projects for an organization. This means that unless project slugs are already known or manually entered, the UI will not show any selectable projects—even if the connection test succeeds with a valid personal access token. The plugin expects to populate the _tool_circleci_projects table with project data, but if no slugs are provided or detected, the list remains empty and the "Save" button is disabled in the UI "project collector", "plugin implementation".

This behavior is similar to issues seen with other integrations (like SonarCloud and Azure DevOps), where missing organization keys or insufficient permissions resulted in empty project lists despite successful connection tests "SonarCloud issue", "Azure DevOps issue". For CircleCI, there is no evidence in the code or documentation that organization-wide project listing is currently supported out of the box.

To troubleshoot, you can enable DEBUG level logging in DevLake to check for backend errors or failed API calls during project listing attempts. As a workaround, you may need to manually enter project slugs or insert them directly into the database until broader project discovery is supported. If you need historical data, webhook connections alone won't suffice.

If you want organization-wide project listing, this would likely require a feature enhancement to the plugin to support fetching all projects for a given organization via the CircleCI API.

To reply, just mention @dosu.

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

Nov 05 '25 13:11 dosubot[bot]

Ok so there's no issue with permissions from CircleCI side. Our stack is Github as the VSC and CircleCI as the CI/CD. It is worth noting also that we connect the two using the Github app.

From a DB point of view:

I had to manually insert the projects into the _tool_circleci_projects table and then i was able to select the projects when adding the connection otherwise it would say no data found so i create the connection and then add the project and then the scope config and currently all circleci tool tables show:

select * from _tool_circleci_accounts; Empty set (0.01 sec)

select * from _tool_circleci_connections; +----+-------------------------+-------------------------+------+---------------------------+-------+---------------------+--------------------------------------------------------------------------------------------------------------+ | id | created_at | updated_at | name | endpoint | proxy | rate_limit_per_hour | token | +----+-------------------------+-------------------------+------+---------------------------+-------+---------------------+--------------------------------------------------------------------------------------------------------------+ | 1 | 2025-11-06 10:52:39.186 | 2025-11-06 10:52:39.186 | CCI | https://circleci.com/api/ | | 0 | aySg3xAv+2F1RhDrVzG2yYbnm6CFYHkjfrf14swPqIOW7x8LcZqPyQjN+rCi5Ru+1qsxz0mlwTcpktxxAURdLb/cJnb1jEQdXTSPlfyWkKk= | +----+-------------------------+-------------------------+------+---------------------------+-------+---------------------+--------------------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec)

select * from _tool_circleci_jobs; Empty set (0.01 sec)

select * from _tool_circleci_pipelines; Empty set (0.00 sec)

select * from _tool_circleci_projects; +---------------+--------------------------------------+--------------------------------------------------------+-------------+--------------------------------------+-----------------+-------------------------+-------------------------+-------------------------------------------------------------------------------------------+----------------------+--------------+------------------+ | connection_id | id | slug | name | organization_id | scope_config_id | created_at | updated_at | _raw_data_params | _raw_data_table | _raw_data_id | _raw_data_remark | +---------------+--------------------------------------+--------------------------------------------------------+-------------+--------------------------------------+-----------------+-------------------------+-------------------------+-------------------------------------------------------------------------------------------+----------------------+--------------+------------------+ | 1 | 4f1a0be6-04b4-4274-a60e-22ea5c7f69ea | circleci/6tYeV1TQWvfmi1b9cxs1mm/AmXx1aZQ9FPmEwAmoogY6u | eir-backend | 2fb020d6-367d-43bf-b9a6-b52340681a44 | 1 | 2025-11-06 10:53:22.031 | 2025-11-06 10:53:56.929 | {"ConnectionId":1,"ProjectSlug":"circleci/6tYeV1TQWvfmi1b9cxs1mm/AmXx1aZQ9FPmEwAmoogY6u"} | _raw_circleci_scopes | 0 | | +---------------+--------------------------------------+--------------------------------------------------------+-------------+--------------------------------------+-----------------+-------------------------+-------------------------+-------------------------------------------------------------------------------------------+----------------------+--------------+------------------+ 1 row in set (0.00 sec)

select * from _tool_circleci_workflows; Empty set (0.01 sec)

Also the cicd deployment tables are currently empty with no data - select * from cicd_deployments; Empty set (0.00 sec) select * from cicd_deployment_commits; Empty set (0.00 sec)

Then i go and collect data and now the tables data have changed as follows: _tool_circleci_accounts; is still empty _tool_circleci_connections; is the same with the cci connection _tool_circleci_jobs; is now populated with 1589 rows _tool_circleci_pipelines; is now populated with 140 rows _tool_circleci_projects; is the same with the project present there _tool_circleci_scope_configs; is the same with the scope config from before _tool_circleci_workflows; is now populated with 139 rows cicd_deployments; is now populated with 105 rows cicd_deployment_commits; is EMPTY

What are we missing to get the cicd_deployment_commits table populated correctly in this case?

It's frustrating that we have a circleci connection option but it just doesn't work it feels half done. I will work with the webhook old approach for now but it'd be amazing if the circleci connection option can be fixed.

Nov 06 '25 12:11 jackwills-op

The most likely reason cicd_deployment_commits is empty is that your deploymentPattern regex in the scope config doesn’t match the actual names of your CircleCI workflows. DevLake uses this pattern to decide which workflows are deployments—if there’s no match, nothing is marked as a deployment, so no deployment commits are generated.

To fix this, check the workflow names in your _tool_circleci_workflows table and compare them to your current deploymentPattern ((?i)^code_build_and_deploy$). If your deployment workflows have different names, update the deploymentPattern in your scope config to match (for example, use a broader regex like (?i)deploy if your workflows contain "deploy" in their names). Once the pattern matches your real deployment workflows, re-run data collection and cicd_deployment_commits should populate as expected.

If you want to confirm the matching logic, see how the plugin uses these patterns to classify deployments in the code. This pattern-based approach is a known usability challenge, so if you have feedback on making it easier, it could help improve the plugin in the future.

To reply, just mention @dosu.

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

Nov 06 '25 12:11 dosubot[bot]

@dosu All the workflow names in the _tool_circleci_workflows table are "code_build_and_deploy" so that all looks fine.

I also checked the code file you sent and see now issue based on the values in _tool_circleci_workflows table but i may be misinterpreting it. Even conditional is never hit here

Nov 06 '25 13:11 jackwills-op

You're right—the conditional in workflow_converter.go only skips workflows with a missing CreatedAt timestamp, so if that's never hit, all your workflows are being processed as expected.

Since your workflow names match the deploymentPattern, the next thing to check is whether your CircleCI jobs (not just workflows) are being classified as deployments. DevLake uses the deploymentPattern to tag jobs as deployment tasks, and if no jobs are marked as deployments, cicd_deployment_commits won't populate—even if the workflows are correct. This tagging depends on the job names matching the deploymentPattern regex, not just the workflow names.

Try running a query like:

SELECT name, type FROM cicd_tasks WHERE type = 'DEPLOYMENT';

If this returns no rows, it means none of your jobs are being marked as deployments. In that case, check the actual job names in your CircleCI runs and adjust your deploymentPattern to match those job names (for example, if your jobs are named "deploy" or "deploy_prod", the pattern should match those).

If you do see deployment tasks, but cicd_deployment_commits is still empty, there may be an issue with how commits are linked to deployments, or a regression in the plugin. Let me know what you find in cicd_tasks and we can dig deeper if needed.

To reply, just mention @dosu.

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

Nov 06 '25 13:11 dosubot[bot]

@dosu You are right, running 'SELECT name, type FROM cicd_tasks WHERE type = 'DEPLOYMENT';' showed no deployment types. I then adjusted the mapping to include all the valid tasks in that table so the regex is now '(?i)(deploy|deployment|code_deploy_ecs)' and then re transformed the data and that made all those tasks have type 'DEPLOYMENT' but still no DORA data and cicd_deployment_commits table is still empty.

What do we look at next?

Side note: I also tried running a fresh collect data command and got this error for circleci run: attached stack trace -- stack trace: | github.com/apache/incubator-devlake/core/runner.RunPluginSubTasks | /app/core/runner/run_task.go:333 | [...repeated from below...] Wraps: (2) subtask collectJobs ended unexpectedly Wraps: (3) attached stack trace -- stack trace: | github.com/apache/incubator-devlake/impls/dalgorm.(*Dalgorm).convertGormError | /app/impls/dalgorm/dalgorm.go:547 | github.com/apache/incubator-devlake/impls/dalgorm.(*Dalgorm).Cursor | /app/impls/dalgorm/dalgorm.go:189 | github.com/apache/incubator-devlake/plugins/circleci/tasks.CollectJobs.func1 | /app/plugins/circleci/tasks/job_collector.go:67 | github.com/apache/incubator-devlake/helpers/pluginhelper/api.NewStatefulApiCollectorForFinalizableEntity | /app/helpers/pluginhelper/api/api_collector_stateful.go:144 | github.com/apache/incubator-devlake/plugins/circleci/tasks.CollectJobs | /app/plugins/circleci/tasks/job_collector.go:49 | github.com/apache/incubator-devlake/core/runner.runSubtask | /app/core/runner/run_task.go:425 | github.com/apache/incubator-devlake/core/runner.RunPluginSubTasks | /app/core/runner/run_task.go:330 | github.com/apache/incubator-devlake/core/runner.RunPluginTask | /app/core/runner/run_task.go:165 | github.com/apache/incubator-devlake/core/runner.RunTask | /app/core/runner/run_task.go:139 | github.com/apache/incubator-devlake/server/services.runTaskStandalone | /app/server/services/task_runner.go:114 | github.com/apache/incubator-devlake/server/services.RunTasksStandalone.func1 | /app/server/services/task.go:187 | runtime.goexit | /usr/local/go/src/runtime/asm_amd64.s:1598 Wraps: (4) Error 1054 (42S22): Unknown column 'created_date' in 'where clause' (500) Wraps: (5) Error 1054 (42S22): Unknown column 'created_date' in 'where clause' Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *mysql.MySQLError

Nov 06 '25 17:11 jackwills-op

@jackwills-op Are you following the projects? IIRC the CircleCI API projects endpoint only returns projects that you follow

Nov 07 '25 11:11 Nickcw6

Thanks @Nickcw6, yes I am already following all my projects -

I got them to show up in devlake after doing db inserts

Nov 07 '25 11:11 jackwills-op