Smoketest with go script

[sago35]

#1330

I created a PR to make it easier to check the CI results. There is no plan to merge it for now.

[sago35]

after: 530fc05

• linux : 29s (4 threads)
  • https://app.circleci.com/pipelines/github/tinygo-org/tinygo/3547/workflows/996f47ee-1831-45d1-b102-e949544700f9/jobs/17753
• macos : 49s (4 threads)
  • https://app.circleci.com/pipelines/github/tinygo-org/tinygo/3547/workflows/996f47ee-1831-45d1-b102-e949544700f9/jobs/17750
• windows : 1m35s (4 threads)
  • https://dev.azure.com/tinygo/tinygo/_build/results?buildId=2081&view=results

[aykevl]

Interesting! This is certainly something that is slow and can be improved by parallelization.

[sago35]

ryzen 3700X (8 core 16 thread)

--threads option	time (real)
16	0m6.397s
15	0m6.494s
14	0m6.655s
13	0m6.636s
12	0m7.263s
11	0m7.177s
10	0m7.428s
9	0m7.541s
8	0m7.917s
7	0m8.467s
6	0m9.219s
5	0m10.524s
4	0m12.322s
3	0m15.590s
2	0m22.600s
1	0m43.584s
original	0m43.519s

[sago35]

The smoketest of tinygo-org/drivers and tinygo-org/tinyfont is also supported.

• tinygo-org/drivers
  • before : 1m27.806s
  • 16 threads : 0m12.605s
    • run-smoketest --threads 16 make smoke-test
• tinygo-org/tinyfont
  • before : 0m46.708s
  • before (-j 16) : 0m14.064s
  • 16 threads : 0m14.0902s
    • run-smoketest --threads 16 make smoketest

note: tinyfont originally supports the make -j

[sago35]

@aykevl @deadprogram I created this PR just as an experiment, but it turns out that tinygo-org/drivers is also faster. The faster smoketest runs, the shorter the feedback loop, which is good.

I'd like to merge into dev branch if possible. Let me know if there are any suggestions.

[sago35]

Rebased.

[sago35]

@aykevl Sometimes it fails. Shouldn't I build them in parallel?

...
tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/espat/espconsole/main.go
   code    data     bss |   flash     ram
  21460    1368    4476 |   22828    5844
2a05b8a7308067c77a7d84213c612858  ./build/test.hex
tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/espat/esphub/main.go
error: rename C:\Users\sago3\AppData\Local\tinygo\pkg-0ad9e123dbad181573f31e6800754f39813d2c803084934e2c099015.bc772198521 C:\Users\sago
3\AppData\Local\tinygo\pkg-0ad9e123dbad181573f31e6800754f39813d2c803084934e2c099015.bc: Access is denied.
tinygo build -size short -o ./build/test.hex -target=arduino-nano33 ./examples/espat/espstation/main.go
   code    data     bss |   flash     ram
  22508    1360    4476 |   23868    5836
dd46c140bdbedf30a516bbc100a36685  ./build/test.hex
...

[sago35]

CI test failed.

windows: https://dev.azure.com/tinygo/tinygo/_build/results?buildId=3109&view=logs&j=ca395085-040a-526b-2ce8-bdc85f692774&t=632617cc-49d1-5f82-37d2-600ea5832b67

build/tinygo build -size short -o test.hex -target=pca10040            examples/adc
error: rename C:\Users\VssAdministrator\AppData\Local\tinygo\pkg-7ce7e135e398bb3f5e0d0966e48407a0a9b3cfde57505737456e56d2.bc213073246 C:\Users\VssAdministrator\AppData\Local\tinygo\pkg-7ce7e135e398bb3f5e0d0966e48407a0a9b3cfde57505737456e56d2.bc: Access is denied.
build/tinygo build -size short -o test.hex -target=pca10040            examples/blinkm
error: rename C:\Users\VssAdministrator\AppData\Local\tinygo\pkg-8ba5a6cc7189b021c75ec0692602d6121b2ca6ce01d98a5f53d172e0.bc133890958 C:\Users\VssAdministrator\AppData\Local\tinygo\pkg-8ba5a6cc7189b021c75ec0692602d6121b2ca6ce01d98a5f53d172e0.bc: Access is denied.
build/tinygo build -size short -o test.hex -target=pca10040            examples/blinky2
error: rename C:\Users\VssAdministrator\AppData\Local\tinygo\pkg-e1813d4acddfd0197001de052beeb17f2721ca3ac59e1a60550b655f.bc397774800 C:\Users\VssAdministrator\AppData\Local\tinygo\pkg-e1813d4acddfd0197001de052beeb17f2721ca3ac59e1a60550b655f.bc: Access is denied.

assert-test-linux: https://app.circleci.com/pipelines/github/tinygo-org/tinygo/4654/workflows/456f3cfa-669e-4aa7-9090-2d5ab3c13767/jobs/26289

#!/bin/bash -eo pipefail
make smoketest TINYGO=build/tinygo

go: downloading github.com/sago35/ochan v0.0.0-20200313013834-e0f46da4579f
go: downloading golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
build/tinygo version
tinygo version 0.18.0-dev-51cea96 linux/amd64 (using go version go1.14.15 and LLVM version 11.0.0)
# test all examples (except pwm)

Too long with no output (exceeded 10m0s): context deadline exceeded

test-llvm10-go113: https://app.circleci.com/pipelines/github/tinygo-org/tinygo/4654/workflows/456f3cfa-669e-4aa7-9090-2d5ab3c13767/jobs/26294

tinygo version 0.18.0-dev linux/amd64 (using go version go1.13.15 and LLVM version 10.0.1)
# test all examples (except pwm)
runtime: failed to create new OS thread (have 52 already; errno=12)
fatal error: newosproc

runtime stack:
runtime.throw(0x4fcdfe, 0x9)
	/usr/local/go/src/runtime/panic.go:774 +0x72
runtime.newosproc(0xc000472380)
	/usr/local/go/src/runtime/os_linux.go:153 +0x1ba
runtime.newm1(0xc000472380)
	/usr/local/go/src/runtime/proc.go:1853 +0xdc
runtime.newm(0x5048c8, 0xc000050000)
	/usr/local/go/src/runtime/proc.go:1832 +0x8f
runtime.startm(0x0, 0xc000605001)
	/usr/local/go/src/runtime/proc.go:1975 +0x12a
runtime.wakep(...)
	/usr/local/go/src/runtime/proc.go:2056
runtime.resetspinning()
	/usr/local/go/src/runtime/proc.go:2436 +0x93
runtime.schedule()
	/usr/local/go/src/runtime/proc.go:2537 +0x2a0
runtime.mstart1()
	/usr/local/go/src/runtime/proc.go:1208 +0x8e
runtime.mstart()
	/usr/local/go/src/runtime/proc.go:1167 +0x6e

goroutine 1 [chan receive (scan), 4 minutes]:
main.run(0xc00010e000, 0x63, 0x80, 0x24, 0x63, 0x80)
	/home/circleci/project/src/cmd/run-smoketest/main.go:113 +0x14b
main.main()
	/home/circleci/project/src/cmd/run-smoketest/main.go:36 +0x201

goroutine 19 [chan receive, 4 minutes]:
github.com/sago35/ochan.NewOchan.func1(0xc0000c23c0)
	/go/pkg/mod/github.com/sago35/[email protected]/ochan.go:34 +0x83
created by github.com/sago35/ochan.NewOchan
	/go/pkg/mod/github.com/sago35/[email protected]/ochan.go:27 +0xcb

[sago35]

When I displayed the runtime.NumCPU() , it was as follows. It seems to be better to set the value at the time of command execution.

• circleci
  • macOS Medium (4 CPU / 8 GB RAM) : runtime.NumCPU() is 4
  • Docker Medium (2 CPU / 4 GB RAM) : runtime.NumCPU() is 36

[sago35]

I figured out why I was getting errors in CI. The problem seems to be that multiple processes are trying to write files to the same location (GOCACHE).

If we pay attention to ioutil.TempFile() and os.Rename(), we can almost solve the problem. The part that has a fixed PATH like compiler-rt-armv7em-non-eabi.a.tmp needs to be addressed somehow.

I will try to change it.

https://app.circleci.com/pipelines/github/tinygo-org/tinygo/4799/workflows/9a0a15f0-9edf-4a07-8953-1b682e6bc970/jobs/27347

error: rename /home/circleci/.cache/tinygo/compiler-rt-armv7em-none-eabi.a.tmp /home/circleci/.cache/tinygo/compiler-rt-armv7em-none-eabi.a: no such file or directory

[sago35]

https://dev.azure.com/tinygo/tinygo/_build/results?buildId=3250&view=logs&j=ca395085-040a-526b-2ce8-bdc85f692774&t=632617cc-49d1-5f82-37d2-600ea5832b67&l=1

error: rename C:\Users\VssAdministrator\AppData\Local\tinygo\dep-577fa6ef79b60e54f3ccab2439e9d74635997471c2fb64ecece730ca.json265915612 C:\Users\VssAdministrator\AppData\Local\tinygo\dep-577fa6ef79b60e54f3ccab2439e9d74635997471c2fb64ecece730ca.json: Access is denied.

[sago35]

I recreated it from the latest dev branch. The CI should fail because the #2348 problem is still there.

The previous version can be found below. https://github.com/tinygo-org/tinygo/tree/smoketest-with-go-script-org

[deadprogram]

The fix for #2348 has been merged into dev branch, just to let you know @sago35

[sago35]

The following worked correctly. I have also confirmed that md5sum matches except for the following

• *.wsm : No problem, as it will always be a different md5sum

On windows bash, the time command can only be used to get real time.

# run-with-smoketest
$ tinygo clean
$ time make smoketest | tee smoketest-new.txt
...
real    1m19.637s
user    0m0.000s
sys     0m0.030s

# original or prev version
$ tinygo clean

$ time make smoketest-commands | tee smoketest-org.txt
...
real    4m13.296s
user    0m0.015s
sys     0m0.093s

[sago35]

Also, so far we've put tools in the tools/ directory. Maybe this one can go there as well? run-smoketest would be the first tool to live in the src directory (the compiler etc are outside of the src directory).

It's better to move it to tools/run-smoketest . I think tinygo users don't need run-smoketest because they don't have the Makefile.

[sago35]

Since md5sum is executed in run-smoketest, it may be better to remove it from the Makefile. What do you think?

[sago35]

rebase. This PR makes the CI a little faster.

os	befer -> after (minutes)	before -> after (seconds)	%
macos	7m50s -> 4m19s	470s -> 259s	55%
linux	6m19s -> 4m20s	379s -> 260s	69%
windows	7m08s -> 5m55s	428s -> 355s	83%

Oops, the following line is not executed. The way it is now, each command needs to have some regularity in smoketest. It seems to be a good idea to put it in make smoketest, but not in make smoketest-commands.

https://github.com/tinygo-org/tinygo/blob/4b2edc9a26a84b4870d61ddf09f70369ab9f6f1e/Makefile#L310-L311

[sago35]

When the Makefile is updated, there is a high probability of conflict. Rebased.