DNM: Combine ceph-pull-requests and API jobs

[djgalloway]

A couple notes:

• I'm open to changing the job name (friendly name and otherwise)
• I'm just disabling the existing/separate make check & API jobs for now. Will delete in the future if we're happy with the new "pipeline"

[epuertat]

@djgalloway as this is causing some fuss in other teams, for the soak testing period, can we just make it run for dashboard-labeled PRs? Thanks!

[djgalloway]

@djgalloway as this is causing some fuss in other teams, for the soak testing period, can we just make it run for dashboard-labeled PRs? Thanks!

Or why don't we just merge as-is and tweak it after the fact?

[djgalloway]

@jdurgin @neha-ojha are y'all okay with merging this PR?

This will disable the ceph-pull-requests and ceph-pr-api jobs in Jenkins. It will change the GitHub Check names to ceph PR - make check and ceph PR - API tests.

The API tests require running make before them so we were essentially running make twice for each PR wasting time and resources.

[epuertat]

@djgalloway something I just realized: and I don't wanna sound like a killjoy, but it'll 'slightly' increment the time taken to finish (10-15 min), as here make check unit tests and API tests are running sequentially. I guess there's no way to run jobs in parallel without saving artifacts?

[djgalloway]

@djgalloway something I just realized: and I don't wanna sound like a killjoy, but it'll 'slightly' increment the time taken to finish (10-15 min), as here make check unit tests and API tests are running sequentially. I guess there's no way to run jobs in parallel without saving artifacts?

I'm sure there probably is. I'm guessing that's where the Pipeline would come in handy. So it'd be:

Phase 1: make Phase 2 and 3 (simultaneous): Unit Tests and API tests ?

[jdurgin]

Sounds good to go to me. We can further improve in future PRs. Reducing the resources by nearly half seems worth merging sooner than later to me.

[neha-ojha]

@jdurgin @neha-ojha are y'all okay with merging this PR?

I have no objections.

This will disable the ceph-pull-requests and ceph-pr-api jobs in Jenkins. It will change the GitHub Check names to ceph PR - make check and ceph PR - API tests.

The API tests require running make before them so we were essentially running make twice for each PR wasting time and resources.

[djgalloway]

Alright, well, I'm on PTO tomorrow so Monday I guess.

[epuertat]

@djgalloway just started to experiment with pipelines (https://github.com/ceph/ceph/pull/37265) and it seems it might not require too much rework...

[djgalloway]

@djgalloway just started to experiment with pipelines (ceph/ceph#37265) and it seems it might not require too much rework...

Okay. Are we going that route instead, then?

[epuertat]

Okay. Are we going that route instead, then?

I guess the multibranch pipeline should be enable on ceph/ceph, and that configuration could still be a jenkins-job-builder YAML file, right? I personally hate the groovy-based syntax of the Jenkinsfile, but I've seen a couple of projects ([1], [2]) to bring YAML support to them.

What do you think?

BTW, I created a pipeline job (epuertat-test-ceph-pipeline), but was... somehow deleted...

[djgalloway]

Okay. Are we going that route instead, then?

I guess the multibranch pipeline should be enable on ceph/ceph, and that configuration could still be a jenkins-job-builder YAML file, right? I personally hate the groovy-based syntax of the Jenkinsfile, but I've seen a couple of projects ([1], [2]) to bring YAML support to them.

What do you think?

My understanding is we'll need a small yaml file defining a pipeline job but that will pull in a groovy Jenkinsfile from ceph.git and ceph-ci.git. I'm fine with that if we can get it working. I didn't feel confident enough in my coding abilities to convert the ceph-pull-requests and ceph-api jobs into groovy so I did it this way.

BTW, I created a pipeline job (epuertat-test-ceph-pipeline), but was... somehow deleted...

Anytime something gets merged into https://github.com/ceph/ceph-build, a Jenkins job runs to update itself and deleted any jobs that aren't defined in this repo.

I would suggest logging in to Jenkins, getting an API key, and using jenkins-job-builder manually to test and upload your JJB YAMLs.

$ cat ~/.jenkins_jobs.jenkins.ceph.com.ini 
[jenkins]
user=GITHUB USERNAME
password=API KEY
url=https://jenkins.ceph.com

Here's the command I used to create/update/test/etc. the ceph-pr-combined job: jenkins-jobs --ignore-cache --conf ~/.jenkins_jobs.jenkins.ceph.com.ini update ceph-pr-combined/config/definitions/ceph-pr-combined.yml

[epuertat]

I would suggest logging in to Jenkins, getting an API key, and using jenkins-job-builder manually to test and upload your JJB YAMLs.

$ cat ~/.jenkins_jobs.jenkins.ceph.com.ini 
[jenkins]
user=GITHUB USERNAME
password=API KEY
url=https://jenkins.ceph.com

Thanks a lot @djgalloway ! Will try it and let you know.

[djgalloway]

This can't be merged as-is anymore. Needs a rebase.

[djgalloway]

@epuertat it pains me to still see us building Ceph binaries twice for every PR. What do you think about merging this next week?

[epuertat]

@epuertat it pains me to still see us building Ceph binaries twice for every PR. What do you think about merging this next week?

@djgalloway omd, completely forgot about this one! I'm totally supportive of this (as a later improvement, we should conditionally include the 'dashboard e2e' tests too). Do you have any recent run of this, to verify it works ok?

[djgalloway]

Do you have any recent run of this, to verify it works ok?

Good call. Will test today.