MIES
MIES copied to clipboard
AllenInstitute
Speedup builds
Description:
Stages are prefixed with -, Jobs with *, and tasks with ~. Bamboo only proceeds to the next stage when all jobs are finished in one stage. Jobs are run in parallel when enough agents are free. Tasks are executed sequentially. Only relevant tasks are shown. BA:XXX gives the bamboo agent capable of executing the job. We currently have three agents (LM, WIN-ITC, WIN-NI).
Plan: Build release (latest)
Branch: release/2.1
- Installer and release packages 5min, BA:WIN-ITC,WIN-NI
- Unit testing 4min, BA:WIN-ITC,WIN-NI
- Uploading 10s, BA:LM
Total (best case): 10min
Plan: Build main
Branch: main
- Create assets
- Documentation (Docker) 12min, BA:LM
- Installer and release package 1min, BA:WIN-ITC,WIN-NI
- Compilation Stage
- Test from repo 12min, BA:WIN-ITC,WIN-NI
- Uploading assets
- Uploading the documentation 20s, BA:LM
- Uploading the installer and ... 30s, BA:LM
Total (best case): 25min
Plan: Checks (Github) new
Branch: All new branches
-
Create artefacts
- CI installer 1min, BA:WIN-ITC,WIN-NI
- Documentation (Docker) 12min, BA:LM
- Installer and release packages 1min, BA:WIN-ITC,WIN-NI
-
Testing without Hardware
- Compilation test installer with hardware XOPs 3min, BA:WIN-ITC,WIN-NI ~ Compilation testing IP8-32bit ~ Compilation testing IP8-64it
- Compilation test installer without hardware XOPs 3min, BA:WIN-ITC,WIN-NI ~ Compilation testing IP8-32bit ~ Compilation testing IP8-64it
- Test from repo (IP8) 20min, BA:WIN-ITC,WIN-NI ~ Compilation testing IP8-32bit (with test scripts) ~ Compilation testing IP8-64bit (with test scripts) ~ Testing without hardware IP8-32bit (RunAllTests.pxp, PA_Tests.pxp) ~ Testing without hardware IP8-64bit (RunAllTests.pxp, PA_Tests.pxp)
- Test from repo (IP9) 12min, BA:WIN-ITC,WIN-NI ~ Compilation testing IP9-32bit (with test scripts) ~ Compilation testing IP9-64bit (with test scripts) ~ Testing without hardware IP9-32bit (RunAllTests.pxp, PA_Tests.pxp) ~ Testing without hardware IP9-64bit (RunAllTests.pxp, PA_Tests.pxp)
-
Hardware Testing
- Testing with ITC18USB hardware 45min (HardwareTests.pxp), BA:WIN-ITC
- Testing with NI hardware 50min (HardwareTests-NI.pxp), BA:WIN-NI
-
Validation
- Validate NWBv2 files 2min, BA:LM
Total (best case): 87min
Ideas for making the builds faster:
- [x] Move the documentation job to the later test stage as it takes much longer than the other tasks in the first stage
- [x] Install a second agent on the linux mint box so that we can parallelize there
- [x] Remove 32bit support from MIES
- Remove submodules: ITCXOP2/ZeroMQ submodules as they are ~ 330MB and need to be downloaded on every job. We can grab what we need for building documentation/installer via curl. We do have the XOPs anyway in XOPs-XXX.
- [x] ZeroMQ
- [x] ITCXOP2
- [ ] Look into Hardware Tests where the time is spent
- [ ] Make stimsets shorter?
- [x] Test from repo (IP8) and Test from repo (IP9) should be only one job
The above numbers are outdated as the bamboo plan was overhauled in 917c09fe (Merge pull request #1494 from AllenInstitute/bugfix/1494-test-reorga-bugfix, 2022-10-03).
Test suite execution times:
rg '<testsuite.*\bname="([^"]+)".*time="([^">]+)".*' --replace '$2 [s]: $1' --no-filename *xml | sort -n
0.111 [s]: UTF_Testpulse.ipf
0.132 [s]: UTF_JSONWaveNotes.ipf
0.183 [s]: UTF_ThreadsafeDataSharing.ipf
0.212 [s]: UTF_UpgradeWaveLocationAndGetIt.ipf
0.223 [s]: UTF_WaveAveraging.ipf
0.243 [s]: UTF_UpgradeDataFolderLocation.ipf
0.566 [s]: UTF_WaveVersioning.ipf
0.894 [s]: UTF_VeryLastTestsuite.ipf
0.930 [s]: UTF_TraceUserData.ipf
1.234 [s]: UTF_AnalysisFunctionHelpers.ipf
1.804 [s]: UTF_EpochswoHardware.ipf
1.862 [s]: UTF_Configuration.ipf
2.114 [s]: UTF_Macros.ipf
2.175 [s]: UTF_StimsetAPI.ipf
2.972 [s]: UTF_PGCSetAndActivateControl.ipf
3.943 [s]: UTF_WaveBuilderRegression.ipf
5.786 [s]: UTF_AsynFrameworkTest.ipf
9.281 [s]: UTF_WaveBuilder.ipf
10.475 [s]: UTF_HardwareTestsWithBUG.ipf
14.814 [s]: UTF_VeryBasicHardwareTests.ipf
18.524 [s]: UTF_AnalysisFunctionParameters.ipf
21.070 [s]: UTF_Utils.ipf
26.745 [s]: UTF_MultiPatchSeqDaScale.ipf
45.560 [s]: UTF_Dashboard.ipf
45.882 [s]: UTF_Labnotebook.ipf
55.181 [s]: UTF_SweepFormula.ipf
59.117 [s]: UTF_ZeroMQPublishing.ipf
65.111 [s]: UTF_ReachTargetVoltage.ipf
66.018 [s]: UTF_AutoTestpulse.ipf
68.411 [s]: UTF_AutoTestpulse.ipf
111.920 [s]: UTF_SweepFormulaHardware.ipf
121.428 [s]: UTF_DAEphys.ipf
132.967 [s]: UTF_PA_Tests.ipf
138.875 [s]: UTF_SweepSkipping.ipf
161.453 [s]: UTF_SetControls.ipf
210.859 [s]: UTF_PatchSeqPipetteInBath.ipf
239.718 [s]: UTF_Epochs.ipf
268.060 [s]: UTF_PatchSeqAccessResistanceSmoke.ipf
279.252 [s]: UTF_PatchSeqSealEvaluation.ipf
319.085 [s]: UTF_PatchSeqTrueRestingMembranePotential.ipf
345.596 [s]: UTF_MultiPatchSeqSpikeControl.ipf
399.543 [s]: UTF_PatchSeqRheobase.ipf
407.057 [s]: UTF_TestPulseAndTPDuringDAQ.ipf
434.365 [s]: UTF_PatchSeqSquarePulse.ipf
487.917 [s]: UTF_PatchSeqRamp.ipf
563.972 [s]: UTF_MultiPatchSeqFastRheoEstimate.ipf
724.704 [s]: UTF_PatchSeqDAScale.ipf
735.203 [s]: UTF_AnalysisFunctionManagement.ipf
919.488 [s]: UTF_TrackSweepCounts.ipf
940.674 [s]: UTF_BasicHardwareTests.ipf
982.640 [s]: UTF_PatchSeqChirp.ipf
The total is 9456s. UTF_AutoTestpulse.ipf is duplicated, fixed in https://github.com/AllenInstitute/MIES/pull/1516.
Test case execution times:
rg '<testcase.*\bname="([^"]+)".*\btime="([^">]+)".*' --replace '$2 [s]: $1' --no-filename *xml | sort -n
The 0.000 for some test cases will be fixed with https://github.com/byte-physics/igor-unit-testing-framework/pull/279.
Taking the results from https://github.com/AllenInstitute/MIES/actions/runs/6152479632:
Non-Hardware tests:
0.001 [s]: @DGEN_SUITE
0.026 [s]: @HOOK_SUITE
0.039 [s]: @HOOK_SUITE
0.174 [s]: UTF_JSONWaveNotes.ipf
0.185 [s]: @DGEN_SUITE
0.919 [s]: UTF_AnalysisFunctionHelpers.ipf
0.975 [s]: UTF_Testpulse.ipf
1.556 [s]: UTF_EpochswoHardware.ipf
2.922 [s]: UTF_PGCSetAndActivateControl.ipf
3.143 [s]: UTF_ThreadsafeDataSharing.ipf
3.792 [s]: UTF_Configuration.ipf
4.396 [s]: UTF_WaveBuilderRegression.ipf
4.599 [s]: UTF_Macros.ipf
4.748 [s]: UTF_UtilsChecks.ipf
4.929 [s]: UTF_AsynFrameworkTest.ipf
5.698 [s]: UTF_AnalysisBrowserTest.ipf
6.160 [s]: UTF_WaveAveraging.ipf
6.314 [s]: UTF_UpgradeDataFolderLocation.ipf
6.354 [s]: UTF_StimsetAPI.ipf
6.976 [s]: UTF_UpgradeWaveLocationAndGetIt.ipf
10.647 [s]: UTF_WaveVersioning.ipf
10.928 [s]: UTF_TraceUserData.ipf
23.225 [s]: UTF_WaveBuilder.ipf
24.331 [s]: UTF_AnalysisFunctionParameters.ipf
42.153 [s]: UTF_DAEphyswoHardware.ipf
46.056 [s]: UTF_Labnotebook.ipf
60.268 [s]: UTF_ZeroMQPublishing.ipf
86.986 [s]: @DGEN_SUITE
97.039 [s]: UTF_SweepFormula_PSX.ipf
98.219 [s]: UTF_HistoricEpochClipping.ipf
103.895 [s]: UTF_HistoricDashboard.ipf
113.181 [s]: UTF_PA_Tests.ipf
127.352 [s]: UTF_SweepFormula.ipf
305.043 [s]: UTF_Utils.ipf
Hardware Tests (ITC):
0.003 [s]: @DGEN_SUITE
0.916 [s]: UTF_VeryLastTestsuite.ipf
0.931 [s]: @DGEN_SUITE
3.504 [s]: @HOOK_SUITE
3.509 [s]: @HOOK_SUITE
9.746 [s]: UTF_Databrowser.ipf
17.546 [s]: UTF_HardwareTestsWithBUG.ipf
19.468 [s]: UTF_VeryBasicHardwareTests.ipf
21.736 [s]: UTF_MultiPatchSeqDaScale.ipf
40.663 [s]: UTF_Dashboard.ipf
56.544 [s]: UTF_ReachTargetVoltage.ipf
56.872 [s]: UTF_AutoTestpulse.ipf
141.277 [s]: UTF_SweepFormulaHardware.ipf
141.653 [s]: UTF_SweepSkipping.ipf
148.292 [s]: UTF_SetControls.ipf
152.901 [s]: UTF_DAEphys.ipf
203.111 [s]: UTF_PatchSeqPipetteInBath.ipf
260.259 [s]: UTF_PatchSeqAccessResistanceSmoke.ipf
267.741 [s]: UTF_PatchSeqSealEvaluation.ipf
286.858 [s]: UTF_MultiPatchSeqSpikeControl.ipf
287.007 [s]: UTF_PatchSeqTrueRestingMembranePotential.ipf
348.587 [s]: UTF_ConfigurationHardware.ipf
363.039 [s]: UTF_PatchSeqRheobase.ipf
374.512 [s]: UTF_PatchSeqSquarePulse.ipf
395.287 [s]: UTF_Epochs.ipf
397.471 [s]: UTF_TestPulseAndTPDuringDAQ.ipf
450.876 [s]: UTF_MultiPatchSeqFastRheoEstimate.ipf
452.627 [s]: UTF_PatchSeqRamp.ipf
640.830 [s]: UTF_PatchSeqDAScale.ipf
672.562 [s]: UTF_AnalysisFunctionManagement.ipf
876.788 [s]: UTF_PatchSeqChirp.ipf
955.336 [s]: UTF_TrackSweepCounts.ipf
1043.198 [s]: UTF_BasicHardwareTests.ipf