[Feature] ArgoCD Integration

[1ddo]

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Use case

Most of our CD are done with Kubernetes ArgoCD and we wanted to add this data connection to DevLake so we could leverage DORA for our deployment frequency.

Description

Add Kubernetes ArgoCD as new data connection to DevLake

Related issues

No response

Are you willing to submit a PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[mintsweet]

hi @1ddo, thank you for your report.

We have received your information and will consider your needs.

[klesh]

@1ddo Hi, I see you checked the Yes I am willing to submit a PR!, would you like to work on it? If so, we could assign the issue to you and you can start working on it.

[1ddo]

Hi @klesh , I'm currently working on it locally, I'm new to this stuff and I'm having difficulty in making devlake work locally in windows. Was able to run the components in Docker Windows however I'm getting API Down in the config UI (localhost:4000), the tutorial said just wait for a few minutes to let the API up and running but mines seems not working. Do you have any more docs on setting devlake locally using windows or any docs that could help me speed things up?

[klesh]

@1ddo Hi, I'm using Windows too because of Feishu(Good collaborative software but lousy Linux support). Sadly, development in Native Windows is not supported, I am using WSL2, which is practically an Ubuntu Linux environment in conjunction with VSCode Remote WSL plugin, you may take a look, it is a far superior development experience than Native Windows.

[1ddo]

Hi @klesh, thanks for the advice. Will try to set up a WSL2

[1ddo]

Hi @klesh , do you know the reason why I don't have a Project menu in my config-ui:4000? I only see Data Connections, Blue Print, Disabled button for Connection and Dashboard menu options?

[abeizn]

@1ddo It should be that your version is not the latest, you can use v0.17.0-beta5 version, the installation method can refer to here: https://devlake.apache.org/docs/next/GettingStarted/DockerComposeSetup

[klesh]

Hi @klesh , do you know the reason why I don't have a Project menu in my config-ui:4000? I only see Data Connections, Blue Print, Disabled button for Connection and Dashboard menu options?

How did you launch devlake backend and config-ui ? Were you using the docker version for the config-ui?

[1ddo]

I've been working all day with the upgrade, by default the version value for the config-ui, grafana and devlake is set to "latest" then I replaced it with the version "v0.17.0-beta5" then ran docker-compose up -d but after that I encountered lots of issues one after another like token, certificate issue, installing grafana cli and others. Till now I'm working with those issues. Did you encounter this issue before? I'm using WSL Ubuntu by the way.

Here's one of the issue I'm getting:

failed to solve: failed to fetch anonymous token: Get "https://auth.docker.io/token?scope=repository%3Alibrary%2Fpython%3Apull&service=registry.docker.io": tls: failed to verify certificate: x509: certificate signed by unknown authority

For the Docker, I've connected my Docker Desktop to WSL Ubuntu and I have the latest Docker Desktop version and was able to run the 4 components (devlake, mysql, config-ui, grafana) but since I'm currently upgrading it to the latest version I haven't tried to run them again since I have a bunch of issues to work on.

[BrookeKatalon]

I'm also interested in the ArgoCD connection

[1ddo]

Hi @BrookeKatalon, are you willing to contribute with the ArgoCD development? If yes then we could collab with it.

[klesh]

@1ddo I'm confused, are you trying to code for devlake or just use it to collect and observe data? For the former, you don't need the docker-compose for bringing up devlake and config-ui, please follow this guide to setup the Development Env https://devlake.apache.org/docs/next/DeveloperManuals/DeveloperSetup For the latter, don't use the docker-compose.yml from the repo, download the one from the release page.

The error message seems like a network problem

[1ddo]

Hi @klesh , we need both since we need collect Data and observed data from ArgoCD but since it's not yet supported then we need to integrate it with DevLake. So don't we need to make DevLake work locally first before start coding, that way we could verify/test it later?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions[bot]]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[klemen-df]

What's the status on this?

[klesh]

@KlemenDanfoss Under discussion, no plan or volunteer at this point AFAIK 😂

[gespi1]

a requirement for the pluginshould be to support argocd multisource applications. Currently devlake takes in one value for repo_url. For argocd apps with mulitple sources the repourl returns a list of multiple repoURL's

[pratiyush05]

Hi @klesh ,we have a scenario with ArgoCD too. We have multiple repos for a single project and then a single team repo . Team repo has image version and from here ArgoCD takes up image versions and do its deployment. So only this team repo's main HEAD commit_sha gets noted in Devlake production deployment from ArgoCD webhook .

We would like to know how Devlake can club multiple repo deployments into a single deployment . In our case , there shall be multiple github repo in a Devlake project and a team repo present in multiple Devlake projects. Plus ArgoCD linked to a single team repo's main branch's HEAD commit only . Could you please suggest a project configuration for us ?

Like clubbing of multiple deployments into one as shown here in blue one .

[klesh]

@pratiyush05 you may create deployment via the webhook plugin , but it actually are creating cicd_deployment_commit records at the moment. We are planning to support multiple repos in the near future by https://github.com/apache/incubator-devlake/issues/6262

[pratiyush05]

Hi @klesh Above diagram shows our current setup . So we would need to add github data connection in this way format - Devlake_Proj_1 - Github Repo_1 + Repo_7 Devlake_Proj_2 - Github Repo_2 + Repo_7 Devlake_Proj_3 - Github Repo_3 + Repo_7 Devlake_Proj_4 - Github Repo_4 + Repo_7 ... total 6 Devlake_projs

My query is how to get deployments of Repo_1 to Repo_6 in their Devlake projects , so that the graph "Median Lead Time to Change" correctly get deployments related to a project . ArgoCD can send commits of Repo_7 only . Since that graph first finds deployments in repos , then the commits and PRs of those commits . So we would need commits from Repo_1 to Repo_6 in Devlake as well .

We are not using Github actions for deployment , so we cannot track deployment in github plugin itself .

[nirmaljeet-singh-groww]

Any idea when can this be releasesd for integration with argocd?

[thiDucTran]

hi, i am willing to help in getting argocd supported. currently, we have a mixed setup. for scm we uses azure repo. for CD, we uses both ArgoCD and Azure DevOps .. with argocd, planning to do https://argocd-notifications.readthedocs.io/en/stable/services/webhook/#send-form-data ..is this the right way to go about it?

[d4x1]

@thiDucTran I think we should add an ArgoCD plugin and collect deployments(and other data) from ArgoCD, then we can combine data from AzureDevops and ArgoCD into one dashboard with some SQL.

[thiDucTran]

i think the issue, for us, is that our ArgoCD auto sync and deploys based on non-application repos (basically whenever argocd see changes to our kustomize overlays)... meaning there needs to be a way to correlate the fact that an ArgoCD deployment was a deployment of which app, which commit of that app etc..

[d4x1]

Can Applications in ArgoCD be related to Azure Repos ?

[YuseopKim]

In my opinion, to use ArgoCD, DevLake needs to make a relationship between deployment repository commit (for manifests) and application repository commit (for source codes).

[klesh]

@YuseopKim We need a separate repo to host ArgoCD stuff, right? We can create one if someone is interested in working on it.

[thiDucTran]

In my opinion, to use ArgoCD, DevLake needs to make a relationship between deployment repository commit (for manifests) and application repository commit (for source codes).

im thinking about the same. one way that i can think of is this flow:

1. CI tool (jenkins, tekton, any CI tool) builds from application repo and produces new docker image. the same CI tool would create a PR to update deployment repo (update manifests to use new docker image); when creating the PR ...we can put the commit hash of the app's repo some where..part of the docker image's tag ..part of the PR's name? this is because when ArgoCD deploys the new docker image ..upon deployment completion, ArgoCD can do some parsing to get the app repo's commit hash and POST it to devlake via webhook?