ROCmSoftwarePlatform
Jenkins CI doesn't carry build cache from last stages
The CI is conservative in reusing build artifacts from last stages. Say benchmarking in 4-th stage can reuse the device instances library from 2nd or 3rd stage, but CI simply deletes all build artifacts and rebuild everything from scratch.
Addressing this issue can reduce CI time and improves our workflow.
I'm a bit confused about the terminology. "Artifacts" are files, e.g., log files, that can be saved at any stage and are stored forever, long after the jenkins process terminates. For example, we are saving the log file of the performance script after the last stage "Performance tests" and you can access it at any later time and use the data from it.
If you want to reuse some of the binaries that were built in the previous stages, that's clearly not possible, because each stage is launched in a separate docker container that gets destroyed once the stage completes.
If it's the latter case, the obvious solution would be to consume every binary/build for test runs in the same docker container where it was built. For example, building ckProfiler used to be one of the steps in the "Static tests" stage previously. When I added the "Performance tests" stage, I just moved it from the "Static tests" stage to the last stage, where we actually use it for running the gemm kernels.
There is also a mechanism of exchanging files between different stages even if they are running in different containers on different physical nodes. You can "stash" files in one stage and "unstash" them in any other stage. This works fine for all sorts of data or log files, but may not be safe for binaries.
So we can totally avoid doing any work twice. It's up to us what to do in the jenkins pipeline and in what order. If you have any suggestions regarding how we could save some time in our current pipeline, @rosenrodt, please let me know and we can look into it.
You are right about the terms. I meant to say preserving build caches. We may use ccache to cache intermediate code objects. ROCm math libs already did.
rocJenkins provides CI config template which specifies ccache configuration
https://github.com/ROCmSoftwarePlatform/rocJenkins/blob/28268bbdd60e1a32ca050af101dda663cf8e91a9/vars/buildProject.groovy#L198-L216
rocBLAS consumes rocJenkins CI configuration and enables ccache by flipping a switch. It seems CK doesn't consume rocJenkins so enabling ccache isn't as straightforward
https://github.com/ROCmSoftwarePlatform/rocBLAS-internal/pull/660