melt-umn
Jenkins takes >30 minutes to do downstream builds of all extensions
I'm about to get Silver building through a Jenkinsfile and integration testing with AbleC, and I find this length of time... annoying. A full Silver integration builds take 46 minutes. Yikes! We should reduce the ablec extensions' build time to less than 15. Ideally less than 5, but that might not be easy.
For the most part, this seems to be an issue with each extension for AbleC taking a rather long time. Even ablec-skeleton takes 6-7 minutes(!!) to run. I'm going to fix this. I'd like feedback on the plan. Do ya'll know what's making things take so long?
A plan:
- [x] I'm going to start working on #55, making extensions use a common "library" for their jenkins build behavior, so I can get things in better shape, and then easily get all the extensions working better.
- [x] Idea 1: I'm going to do the initial build with
--warn-all --warn-error, and skip the MWDA and only do the MDA during the later step of modular analyses. This avoids one full rebuild in Jenkins. - [x] Idea 2: I want to reduce the amount of re-building, so I may make sure the MDA can be run without needing the
--cleanflag. - [x] Idea 3: I may add a parameter that tells the extension jenkinsfile how to get ablec's host build's generated files, so we can avoid re-building even that when ablec was just built.
- [x] Idea 4: I think the test makefile is rebuilding the extended jar? We should have it use the one already built...
- [x] Idea 5: A better approach to checking for a branch's existence than the exception approach we have right now.
- I'm mildly annoyed by having to build so many different parsers in AbleC itself, but that may be unavoidable for now...? (Although... is decl separate from decls necessary? That would reduce the number by 20%!)
- (I wish we could reduce the memory usage of silver builds, then we could increase executor count, but I don't know if there's any low hanging fruit there.)
Hi @krame505 @TravisCarlson do either of you have thoughts on what else might be taking so long or comments about the above ideas?
Idea 1: I'm going to do the initial build with --warn-all --warn-error, and skip the MWDA and only do the MDA during the later step of modular analyses. This avoids one full rebuild in Jenkins.
Reasonable, I guess, but part of the goal is to fully test the Makefiles as well, which would include the commands for performing the analyses. Not too big of a deal I guess...
Idea 2: I want to reduce the amount of re-building, so I may make sure the MDA can be run without needing the --clean flag. Idea 3: I may add a parameter that tells the extension jenkinsfile how to get ablec's host build's generated files, so we can avoid re-building even that when ablec was just built.
Sure.
Idea 4: I think the test makefile is rebuilding the extended jar? We should have it use the one already built...
I'd need to dig up the old thread, but I remember us having rather compelling reasons why we wanted to keep examples and tests totally independent. But if we can avoid doing a clean build between building examples and building tests, this second build should be very quick anyway.
Idea 5: A better approach to checking for a branch's existence than the exception approach we have right now.
Agreed, although this isn't a significant contributor to the performance problems since the error happens right away.
I'm mildly annoyed by having to build so many different parsers in AbleC itself, but that may be unavoidable for now...? (Although... is decl separate from decls necessary? That would reduce the number by 20%!)
Decl is seperate from Decls since there are times when you want to parse only one Decl, and I guess this saves you having to wrap the result in decls(...)... so not really that important I guess. But shouldn't this only be getting built when we build an extension that actually imports the construction:parsing library? This shouldn't be getting exported...
Anyway the silver-ableC extension should (soonish) render this essentially deprecated - although that will add its own build time penalty :-P
this second build should be very quick anyway.
It still ends up taking a rather remarkable amount of time. Like 35 seconds or so. The process of building a jar file from soooo many little files takes awhile. :/
but part of the goal is to fully test the Makefiles as well
Fair enough... new plan:
- [x] Idea 6: I'll have a "quick build" jenkins parameter that favors turning on these go-fast alternatives, so regular builds (i.e. changes to the extension) will still fully test the makefile in the current manner. Actually, maybe I'll just re-use to the "here's host generated files" parameter. If you get that, go fast, the extension hasn't changed only host/silver, if you don't, be thorough.
since the error happens right away.
Unless the error is something other than branch existence, in which case it tries building more than once! :)
skeleton fast build times reduced from 6.5 minutes to 4 minutes.
all extensions build time reduced from 31 minutes to 25 minutes. Possibly this number is a slightly high fluke for some random reason, even.
Remaining known tasks:
- Unchecked above: removing --clean from doing the MDA. Requires changing makefiles. Should be fine to remove always, since there's already a
touch... - [x] Copying generated files seems to still make copper rebuild all parsers. I committed a fix for this, but haven't yet verified that it's working, nor included in numbers above.
A quick note about something I noticed in makefiles in these extensions: some of them seem "out of date"
I saw old-style makefiles while looking through:
- ableC-cilk
- ableC-condition-tables
- ableC-sqlite
Eric has been hesitant to adopt Makefiles with any sort of generic or pattern rules since he thinks they are harder to read. I am in favor of using the generic Makefiles everywhere, for the sake of maintainability. The result is that the extensions that I primarily develop have the new-style Makefiles, while some of the other extensions have hardcoded Makefiles. I am still in favor of moving everything to the generic Makefiles (annoying to fix each nonstandard Makefile when something breaks!) but this is kind of up to Eric.
Another general performance improvement for ableC Jenkins builds:
- [x] Build ableC tutorials in parallel
All extensions total build time is reduced to 16 minutes, which seems rather successful.
- [x] Lucas suggests not translating / building jars when doing e.g. the MDA in https://github.com/melt-umn/silver/issues/248
- [x] I think https://github.com/melt-umn/silver/issues/232 allowing us to copy around the AbleC jar instead of the whole AbleC generated directory would be a dramatic speed up as well
The above happened, but we're not caching a silver-ableC.jar so we're back at a slow build (on a different, almost certainly slower-per-core machine). I'm not bothered by it much (esp. since GitHub got "queue this merge," but the last success at time of writing was 41min.
The next blocking issue might be
- https://github.com/melt-umn/copper/issues/62
?
Note that the "diamond dependency" issue means that it's not reasonable to approach this by making extensions actually build downstream from their dependencies.
A reasonable fix for this is probably to make the ableC extensions triggered downstream of Silver in a single Jenkins workspace, thus caching the artifacts between these builds. This would require use of flock or something when building an artifact, to prevent multiple concurrent builds being triggered for an artifact.
After the latest CPU upgrade, we're down to 27min for the ableC build, so in principle we can close this :)
Yeah, I think we can close this for now. deep-rebuild is slow now with the move to separate artifacts, but that's not specific to Jenkins.