sysprog21
Fix benchmark CI pipeline
Recompile dhrystone.elf and coremark.elf in order to fix the benchmark CI pipeline, as discussed at https://github.com/sysprog21/rv32emu/issues/455#issuecomment-2163491139
References:
- https://github.com/sysprog21/rv32emu/issues/451
- https://github.com/sysprog21/rv32emu/issues/455
@jserv do you still have the documentation/Makefile/etc. for compiling coremark.elf?
do you still have the documentation/Makefile/etc. for compiling
coremark.elf?
Run git log build/coremark.elf to figure out the steps to rebuild CoreMark.
do you still have the documentation/Makefile/etc. for compiling
coremark.elf?Run
git log build/coremark.elfto figure out the steps to rebuild CoreMark.
Replaced coremark using the binary compiled from rv32emu-prebuilt
@jserv The performance is still slow after replacing both binaries (using the ones compiled from rv32emu-prebuilt)
@jserv The performance is still slow after replacing both binaries (using the ones compiled from
rv32emu-prebuilt)
@vacantron, Can you check?
Due to GitHub CI pipeline security protection (it's my guess), my latest changes aren't visible here. Please check https://github.com/henrybear327/rv32emu/pull/5.
@vacantron thanks for providing the arguments to be used for coremark execution! Though I am not sure of the root cause of this error when executing Coremark, cc @jserv
rv32emu: src/emulate.c:632: block_translate: Assertion `prev_ir' failed.
Aborted (core dumped)
Start Test CoreMark benchmark
Traceback (most recent call last):
File "/home/runner/work/rv32emu/rv32emu/tests/coremark.py", line 16, in <module>
comp_proc = subprocess.check_output(
File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'build/rv32emu build/coremark.elf 0x0 0x0 0x66 60000 7 1 [20](https://github.com/henrybear327/rv32emu/actions/runs/9846674893/job/27184952365?pr=5#step:5:21)00' returned non-zero exit status 134.
Traceback (most recent call last):
File "/home/runner/work/rv32emu/rv32emu/tests/bench-aggregator.py", line 25, in <module>
run_benchmark(b)
File "/home/runner/work/rv32emu/rv32emu/tests/bench-aggregator.py", line 13, in run_benchmark
subprocess.run(args=[interp, b], shell=False, check=True)
File "/usr/lib/python3.10/subprocess.py", line 5[26](https://github.com/henrybear327/rv32emu/actions/runs/9846674893/job/27184952365?pr=5#step:5:27), in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python3', 'tests/coremark.py']' returned non-zero exit status 1.
Regarding Dhrystone, that DMIPS value is now reported normally!
Though I am not sure of the root cause of this error when executing Coremark
The Coremark with rv32imafc extension should work normally now. It seems there are some problems with the certain extension combination.
It is confusing to have
benchmark.ymlandbenchmark-new.yml.
Synced offline with @jserv, this is a temporary testing method.
A new discovery is that if I move the benchmark pipeline to main.yml, the latest changes on the pipeline can be reflected directly with the PR.
I currently have no explanation for this, but to speed up development I will keep the benchmark pipeline in main.yml for now.
I currently have no explanation for this, but to speed up development I will keep the benchmark pipeline in
main.ymlfor now.
It sounds acceptable. Let's get the benchark work again.
Good news! The pipeline is finally back online with normal numbers again! Coremark is currently compiled using rv32im and rv32imafc (thanks to @vacantron for the suggestion). Both can be executed without failure!
@jserv do you happen to remember what were the parameters you used for Coremark? It's not found within the past commit messages but we would need to supply this in the benchmark script. Currently, the CI is using 0x0 0x0 0x66 30000 7 1 2000 (>= 15000 iterations should be more than 10s of execution time)
Thank you!
Though I am not sure of the root cause of this error when executing Coremark
The Coremark with
rv32imafcextension should work normally now. It seems there are some problems with the certain extension combination.
I have tried rv32im (https://github.com/sysprog21/rv32emu/commits/master/build/coremark.elf), also works!
do you happen to remember what were the parameters you used for Coremark? It's not found within the past commit messages but we would need to supply this in the benchmark script. Currently, the CI is using
0x0 0x0 0x66 30000 7 1 2000(>= 15000 iterations should be more than 10s of execution time)
Prior to this pull request, there was no parameter passing to CoreMark.
do you happen to remember what were the parameters you used for Coremark? It's not found within the past commit messages but we would need to supply this in the benchmark script. Currently, the CI is using
0x0 0x0 0x66 30000 7 1 2000(>= 15000 iterations should be more than 10s of execution time)Prior to this pull request, there was no parameter passing to CoreMark.
I took the arguments from https://github.com/HidetaroTanaka/coremark/blob/main/Makefile#L103 and https://github.com/HidetaroTanaka/coremark/blob/main/Makefile#L107.
According to the experiment, executing coremark for 1 second can perform around 1200 iterations. Also, if the execution time is less than 10s, coremark will fail with a non-zero exit code. So I set the iteration to 20000 for now.
Known issues:
- Benchmark / Performance regression check: it's already deleted, but still being executed
- CI / benchmark: can't access
secrets.RV32EMU_BENCH_TOKEN(I believe after merging the PR this will be resolved, as this is a security feature so we can't leak the secret using a PR from a contributor)
Move the commit "Fix typo and remove outdated comment" out of this pull request.
Move the commit "Fix typo and remove outdated comment" out of this pull request.
Done as requested (a new PR is created)
Move the commit "Fix typo and remove outdated comment" out of this pull request.
Done as requested (a new PR is created)
Since there are no code changes, the CI pipeline won't be triggered, thus, passing all of them!