sp00n
Enhancement: Allow "chaining" of configs
Suggestion and some background Background: I've ran nearly 600 hours with CoreCycler now for 9950X3D after setting up a boost override increase and adding negative curve optimizer voltage.
First set of tests included a bunch of different configs to have varying load types and to get rid of all crashes/instabilities while fine tuning the curve offset for each individual core. This meant a lot of observing, adjusting, and where most of the time was spent, as expected.
Now that the system has passed through most of the tests and I'm not expecting it to crash or even throw an error at this point; I am doing one final round of tests (around 10 different configs) for X amount of iteration and then loading up a new one.
It would've been nice to be able to write in the original config to just continue running the session after the iteration is done, but using the new config settings. Because right now, I'm essentially waiting for it to finish some iterations, then restart corecycler after pointing to a new config, and repeat.
I.e.
ContinueAfterIterationCompletion = config1,config2,config3,config4
This way one could just "queue" a bunch of different configs and let it continue going through each config without having to observe it or relaunch things along the way.
I'm not sure what the best approach is; I can imagine that keeping the session alive and just "reading" the config would be better because then it doesn't just "forget" that it is supposed to continue on in the next config (if it fully restarted/relaunched), or requiring more flags in different configs.
Some information regarding stability testing 9950X3D (or Ryzen 9000), to whoever might find it interesting
- Make sure RAM/EXPO is stable first and foremost
- 1 Thread or 2 Thread has big impact in terms of detecting stability issues, even if Test X seemed stable on 1 thread (you should test both)
- "suspendPeriodically" flag has a big impact in terms of detecting stability issues (you should test with it enabled and disabled)
- KOMARI was the quickest to find very unstable cores or unstable RAM; to identify where to make adjustments
- KAGARI was the test that found the most errors/instability over time (4 times on 1 thread, 11 times on 2 threads)
- Followed by MIYU and 00-X86 depending on setting (1T vs 2T, suspend vs not suspend)
- Heavily using your PC will trigger 'CPU LOAD' errors unless you increase the Priority setting under [Debug]; I recommend leaving PC alone or do very light work while it tests in peace to avoid these "false positive" errors. That way, if it does occur, you know it is a true positive error
Wanted to make sure the system was stable in the following situations:
- AVX512, AVX2, AVX; full load and single core load (single and multi threaded)
- FMA3, SSE and light loads; full load and single core load (single and multi threaded)
Test used (Settings used) = Total run time for test:
KOMARI (1T Suspend On + 2T Suspend Off) = 70~ hours KAGARI (1T Suspend On + 2T Suspend Off) = 78~ hours MIYU (1T Suspend On + 2T Suspend Off) = 77~ hours HINA (1T Suspend On + 2T Suspend Off) = 74~ hours 04-P4P (1T Suspend On + 2T Suspend Off) = 88~ hours 00-x86 (1T Suspend On + 2T Suspend Off) = 97~ hours
Prime95 AVX512 (1T 6 min runtime Suspend On + 2T auto runtime Suspend Off) (ALL FFTs) = 50~ hours Prime95 AVX2 (1T 6 min runtime Suspend On + 2T auto runtime Suspend Off) (ALL FFTs) = 66~ hours Prime95 AVX (1T 6 min runtime Suspend On + 2T auto runtime Suspend Off) (ALL FFTs) = 50~ hours Prime95 SSE (1T 6 min runtime Suspend On + 2T auto runtime Suspend Off) (ALL FFTs) = 64~ hours
Prime95 AVX512 Full Load - 1T & 2T (launched separately) = 21~ hours Prime95 AVX2 Full Load - 1T & 2T (launched separately) = 7~ hours Prime95 AVX Full Load - 1T & 2T (launched separately) = 7~ hours Prime95 SSE Full Load - 1T & 2T (launched separately) = 14~ hours
Finished everything with a round of short, bursty, 1x iteration multi-config run (3m runtime, 16 cores = 50min~ per iteration/test):
"KOMARI", 2T, Suspend Off "KOMARI", 2T, Suspend On "KOMARI", 1T, Suspend On "KOMARI", 1T, Suspend Off "KOMARI", 1T, Suspend Off + AssignBothVirtualCores "KAGARI", 2T, Suspend Off "KAGARI", 2T, Suspend On "KAGARI", 1T, Suspend On "KAGARI", 1T, Suspend Off "KAGARI", 1T, Suspend Off + AssignBothVirtualCores "MIYU", 2T, Suspend Off "MIYU", 2T, Suspend On "MIYU", 1T, Suspend On "MIYU", 1T, Suspend Off "MIYU", 1T, Suspend Off + AssignBothVirtualCores "HINA", 2T, Suspend Off "HINA", 2T, Suspend On "HINA", 1T, Suspend On "HINA", 1T, Suspend Off "HINA", 1T, Suspend Off + AssignBothVirtualCores 04-P4P, 2T, Suspend Off 04-P4P, 2T, Suspend On 04-P4P, 1T, Suspend On 04-P4P, 1T, Suspend Off 04-P4P, 1T, Suspend Off + AssignBothVirtualCores 00-x86, 2T, Suspend Off 00-x86, 2T, Suspend On 00-x86, 1T, Suspend On 00-x86, 1T, Suspend Off 00-x86, 1T, Suspend Off + AssignBothVirtualCores
And a last one, 28x bursty multi-config with 1min runtime with every test, just for good measure (16-17min per iteration/test):
"KOMARI", 1T+2T Suspend Off + AssignBothVirtualCores "KIZUNA", 1T+2T Suspend Off + AssignBothVirtualCores "KAGARI", 1T+2T Suspend Off + AssignBothVirtualCores "SHINOA", 1T+2T Suspend Off + AssignBothVirtualCores "YUKINA", 1T+2T Suspend Off + AssignBothVirtualCores "KOTORI", 1T+2T Suspend Off + AssignBothVirtualCores "KURUMI", 1T+2T Suspend Off + AssignBothVirtualCores "AIRI", 1T+2T Suspend Off + AssignBothVirtualCores "MIYU", 1T+2T Suspend Off + AssignBothVirtualCores "HINA", 1T+2T Suspend Off + AssignBothVirtualCores "USHIO", 1T+2T Suspend Off + AssignBothVirtualCores "KASUMI", 1T+2T Suspend Off + AssignBothVirtualCores 04-P4P, 1T+2T Suspend Off + AssignBothVirtualCores 00-X86, 1T+2T Suspend Off + AssignBothVirtualCores
~I'm in the final step of finishing this up (which lead to me suggesting this enhancement), but soon clocking in 600 hours of pure testing (running test programs), excluding administrative tasks or troubleshooting and other time costs.~
Finished this up just under 800 hours of pure testing curve optimizer stability for 9950X3D - excluding administrative task/trouble shooting and other time costs.
What I could think of right now would be to create a custom batch file to run the script, and then if it exits without an error, copy a different config file to the config.ini location and re-start the script with this new config.
I think this should work, but I'd first have to add an exit / error code to the script when it does not end gracefully and make some tests to check if this actually works as I imagine.
You could try this batch file, which looks for multiconfig-x.ini files, and processes them sequentially if the script doesn't quit with an error.
Either a script error will stop that, or if all of the cores have thrown an error during one config file run, or if the stopOnError setting has been enabled.
It doesn't really work with the Automatic Test Mode, at least not with the resuming functionality after a crash. In that case the resume script will just load the script as usually with the config that was tested last, and ignore any multiconfig files.
@echo off
SETLOCAL EnableDelayedExpansion
REM This uses the format "multiconfig-x.ini", so e.g. multiconfig-1.ini, multiconfig-2.ini, multiconfig-3.ini
echo Starting a multiconfig run of CoreCycler...
REM Prepare the variable for the last config file
SET "LASTCONFIG="
FOR /F %%G IN ('dir /b multiconfig-*.ini') DO SET "LASTCONFIG=%%G"
REM Loop through the config files
FOR /R %%G IN (multiconfig-*.ini) DO (
echo.
echo ================================================================================
echo [!TIME!] Starting run with %%~nxG
echo ================================================================================
copy /Y %%G config.ini >nul
IF !ERRORLEVEL! NEQ 0 (
echo ERROR^^^! Could not find %%G
pause
exit
)
echo.
REM Doesn't really work with the Automatic Test Mode yet
REM start "CoreCycler" cmd.exe /k powershell.exe -ExecutionPolicy Bypass -File "%~dp0script-corecycler.ps1" %PARAMS%
REM Extra command for the last iteration, to keep the window open
IF !LASTCONFIG! EQU %%~nxG (
echo This is the last config
cmd.exe /k powershell.exe -ExecutionPolicy Bypass -File "%~dp0script-corecycler.ps1"
) ELSE (
powershell.exe -ExecutionPolicy Bypass -File "%~dp0script-corecycler.ps1"
)
IF !ERRORLEVEL! NEQ 0 (
echo Error level: !ERRORLEVEL!
echo [!TIME!] ERROR^^^! CoreCycler run did not complete successfully^^^!
pause
exit !ERRORLEVEL!
)
echo.
)
Tested this but it cannot find the config files even though the path is correct.
You should just place the multiconfig-1.ini, multiconfig-2.ini etc inside the main directory and then run the batch file. A "path" shouldn't have any part in this.
And I forgot to mention, you should also use the current script-corecycler.ps1 from the repository, it contains the exit codes when something has gone wrong, so that the batch file stops and doesn't just continue with the next config.
You should just place the multiconfig-1.ini, multiconfig-2.ini etc inside the main directory and then run the batch file.
I did this and the script returns error that it cannot find the config, I placed 4 of them in the main directory as written.
Copy pasting the path it provides in the error message, in explorer, opens the config (meaning the config is there)
Can you make a screenshot of the Explorer window with your CoreCycler installation?
That's the current version, which does work with spaces:
@echo off
SETLOCAL EnableDelayedExpansion
REM --------------------------------------------------------------------------------------
REM This batch file can be used to run CoreCycler with multiple different configs in a row
REM The config files need to be in the format "multiconfig-x.ini",
REM so e.g. multiconfig-1.ini, multiconfig-2.ini, multiconfig-3.ini, etc.
REM
REM The batch file will continue to the next config file as long as the current config run
REM ends normally.
REM It will stop if the script either throws a script error, e.g. something has gone
REM terribly wrong, all of the cores have thrown an error during the current config run,
REM or if the "stopOnError" setting has been enabled in the config file.
REM --------------------------------------------------------------------------------------
echo Starting a multiconfig run of CoreCycler...
REM Prepare the variable for the last config file
SET "LASTCONFIG="
FOR /F %%G IN ('dir /b multiconfig-*.ini') DO SET "LASTCONFIG=%%G"
IF [!LASTCONFIG!] EQU [] (
echo Error^^^! No multiconfig files found, aborting^^^!
pause
exit 1
)
REM Loop through the config files
FOR /R %%G IN (multiconfig-*.ini) DO (
SET "CURRENTCONFIG=%%~nxG"
echo.
echo ================================================================================
echo [!TIME!] Starting run with !CURRENTCONFIG!
echo ================================================================================
copy /Y "!CURRENTCONFIG!" config.ini >nul
IF !ERRORLEVEL! NEQ 0 (
echo ERROR^^^! Could not find "!CURRENTCONFIG!"
pause
exit 2
)
echo.
REM Doesn't really work with the Automatic Test Mode yet
REM start "CoreCycler" cmd.exe /k powershell.exe -ExecutionPolicy Bypass -File "%~dp0script-corecycler.ps1" %PARAMS%
REM Extra command for the last iteration, to keep the window open
IF !LASTCONFIG! EQU !CURRENTCONFIG! (
echo This is the last config
cmd.exe /k powershell.exe -ExecutionPolicy Bypass -File "%~dp0script-corecycler.ps1"
) ELSE (
powershell.exe -ExecutionPolicy Bypass -File "%~dp0script-corecycler.ps1"
)
IF !ERRORLEVEL! NEQ 0 (
echo Error level: !ERRORLEVEL!
echo [!TIME!] ERROR^^^! CoreCycler run did not complete successfully^^^!
pause
exit !ERRORLEVEL!
)
echo.
)
pause
I haven't error tested this much; but as for it running through the configs sequentially, it is doing that and it works good and all the output remains in the same window so it's very easy to see what has happened!
Much appreciated for this script!
Used this quite a bit now and it's working fine! One minor detail is that if you use 10+ configs, it's going to do # 1 and then all the 10ths before carrying on with 2, 3, 4 etc, so not entirely sequential.
Ah, the good old alphabetical versus numerical ordering.
You should be able to use multiconfig-01.ini, multiconfig-02.ini, ... multiconfig-99.ini to get around this.
Actually you could also use a-z instead of numbers, it's just looking for anything that matches multiconfig-*.ini.
@sp00n The batch script doesn't work if you try to run it as Administrator by the way. I guess that's not an issue since it's not used with the Automatic Testing anyway but just thought I'd mention it.
Thanks for the writeup @NoSkills. I am going to try a similar setup with my 7900x, albeit scaled back considerably. I am not willing to do 2-3 day long tests 😄!! Maybe when I get a new system down the line and can run these while still using my current computer in the meantime.
Starting as admin will set the directory of the terminal window to C:\Windows\system32, so it won't find the ini files.
You can add pushd %~dp0 right after echo Starting a multiconfig run of CoreCycler..., which made it work for me.
@sp00n The batch script doesn't work if you try to run it as Administrator by the way. I guess that's not an issue since it's not used with the Automatic Testing anyway but just thought I'd mention it.
Thanks for the writeup @NoSkills. I am going to try a similar setup with my 7900x, albeit scaled back considerably. I am not willing to do 2-3 day long tests 😄!! Maybe when I get a new system down the line and can run these while still using my current computer in the meantime.
No worries :) I gotta say that Automatic Testing seems like a godsend for this. 9000-series wasn't supported in corecycles for automatic, so I did it manually - every crash/hang was a trip to bios, make adjustments, rerun test, and waking up during the night just to see if things had hanged or not.
Luckily, I got it reasonably stable kind of early and I believe with some insights above + multiconfig chaining, one can create very bursty/crash-prone testing and get the system stable much earlier and without having to observe the PC too much; even less so with automatic testing.