Running Julia tests

[goerch]

What is the recommended way to run juliatests.jl on VS Code?

[goerch]

I would like to `include("test/juliatests.jl"). After adapting the paths I now run into the following exceptions:

ambiguous => RemoteException(1, CapturedException(CompositeException(Any[CapturedException(ErrorException("Error deserializing a remote exception from worker 13\nRemote(original) exception of type MethodError\nRemote stacktrace : "), Any[(top-level scope at utils.jl:189, 1), (eval at boot.jl:368 [inlined], 1), (run_test_by_eval at utils.jl:175, 1), (#invokelatest#2 at essentials.jl:731, 1), (invokelatest at essentials.jl:729, 1), (#106 at process_messages.jl:278, 1), (run_work_thunk at process_messages.jl:63, 1), (macro expansion at process_messages.jl:278 [inlined], 1), (#105 at task.jl:466, 1)]), CapturedException(UndefVarError(:JuliaTests), Any[(deserialize_module(s::Distributed.ClusterSerializer{TCPSocket}) at Serialization.jl:995, 1), (handle_deserialize(s::Distributed.ClusterSerializer{TCPSocket}, b::Int32) at Serialization.jl:894, 1), (deserialize(s::Distributed.ClusterSerializer{TCPSocket}) at Serialization.jl:812, 1), ((::Serialization.var"#5#6"{Distributed.ClusterSerializer{TCPSocket}})(i::Int64) at Serialization.jl:971, 1), (ntupleany(f::Serialization.var"#5#6"{Distributed.ClusterSerializer{TCPSocket}}, n::Int64) at ntuple.jl:43, 1), (deserialize_tuple(s::Distributed.ClusterSerializer{TCPSocket}, len::Int64) at Serialization.jl:971, 1), (handle_deserialize(s::Distributed.ClusterSerializer{TCPSocket}, b::Int32) at Serialization.jl:855, 1), (deserialize(s::Distributed.ClusterSerializer{TCPSocket}) at Serialization.jl:812, 1), ((::Serialization.var"#5#6"{Distributed.ClusterSerializer{TCPSocket}})(i::Int64) at Serialization.jl:971, 1), (ntupleany(f::Serialization.var"#5#6"{Distributed.ClusterSerializer{TCPSocket}}, n::Int64) at ntuple.jl:43, 1), (deserialize_tuple(s::Distributed.ClusterSerializer{TCPSocket}, len::Int64) at Serialization.jl:971, 1), (handle_deserialize(s::Distributed.ClusterSerializer{TCPSocket}, b::Int32) at Serialization.jl:855, 1), (deserialize(s::Distributed.ClusterSerializer{TCPSocket}, t::DataType) at Serialization.jl:1464, 1), (handle_deserialize(s::Distributed.ClusterSerializer{TCPSocket}, b::Int32) at Serialization.jl:876, 1), (deserialize at Serialization.jl:812 [inlined], 1), (deserialize(s::Distributed.ClusterSerializer{TCPSocket}, t::Type{CapturedException}) at clusterserialize.jl:223, 1), (handle_deserialize(s::Distributed.ClusterSerializer{TCPSocket}, b::Int32) at Serialization.jl:876, 1), (deserialize(s::Distributed.ClusterSerializer{TCPSocket}, t::DataType) at Serialization.jl:1464, 1), (handle_deserialize(s::Distributed.ClusterSerializer{TCPSocket}, b::Int32) at Serialization.jl:876, 1), (deserialize at Serialization.jl:812 [inlined], 1), (deserialize_msg(s::Distributed.ClusterSerializer{TCPSocket}) at messages.jl:87, 1), (#invokelatest#2 at essentials.jl:731 [inlined], 1), (invokelatest at essentials.jl:729 [inlined], 1), (message_handler_loop(r_stream::TCPSocket, w_stream::TCPSocket, incoming::Bool) at process_messages.jl:169, 1), (process_tcp_streams(r_stream::TCPSocket, w_stream::TCPSocket, incoming::Bool) at process_messages.jl:126, 1), ((::Distributed.var"#99#100"{TCPSocket, TCPSocket, Bool})() at task.jl:466, 1)])]), Any[(deserialize(s::Distributed.ClusterSerializer{TCPSocket}, t::Type{CapturedException}) at clusterserialize.jl:225, 1), (handle_deserialize(s::Distributed.ClusterSerializer{TCPSocket}, b::Int32) at Serialization.jl:876, 1), (deserialize(s::Distributed.ClusterSerializer{TCPSocket}, t::DataType) at Serialization.jl:1464, 1), (handle_deserialize(s::Distributed.ClusterSerializer{TCPSocket}, b::Int32) at Serialization.jl:876, 1), (deserialize at Serialization.jl:812 [inlined], 
1), (deserialize_msg(s::Distributed.ClusterSerializer{TCPSocket}) at messages.jl:87, 1), (#invokelatest#2 at essentials.jl:731 [inlined], 1), (invokelatest at essentials.jl:729 [inlined], 1), (message_handler_loop(r_stream::TCPSocket, w_stream::TCPSocket, incoming::Bool) at process_messages.jl:169, 1), (process_tcp_streams(r_stream::TCPSocket, w_stream::TCPSocket, incoming::Bool) at process_messages.jl:126, 1), ((::Distributed.var"#99#100"{TCPSocket, TCPSocket, Bool})() at task.jl:466, 1)]))```

[goerch]

Hm, after some changes I got that working:

Julia Version 1.7.0
Commit 3bf9d17731 (2021-11-30 12:12 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-10710U CPU @ 1.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake)
Environment:
  JULIA_EDITOR = code
  JULIA_NUM_THREADS = 5
Test run at: 2021-12-10T11:06:04.727

Maximum number of statements per lowered expression: 10000

I excluded ranges because of problems...

Test file	Passes	Fails	Errors	Broken	Aborted blocks
ambiguous	34	1	3	0	15
compiler/inference	X	X	X	X	X
compiler/validation	17	0	5	0	8
compiler/ssair	X	X	X	X	X
compiler/irpasses	X	X	X	X	X
compiler/codegen	X	X	X	X	X
compiler/inline	X	X	X	X	X
compiler/contextual	5	1	6	0	8
subarray	238	0	21	0	16
strings/basic	152	0	1	0	17
strings/search	X	X	X	X	X
strings/util	615	0	0	0	7
strings/io	12735	0	0	0	4
strings/types	2302691	0	0	0	5
unicode/utf8	19	0	0	0	1
core	X	X	X	X	X
worlds	X	X	X	X	X
atomics	X	X	X	X	X
keywordargs	5	0	11	0	20
numbers	X	X	X	X	X
subtype	X	X	X	X	X
char	1524	0	2	0	4
triplequote	29	0	0	0	0
intrinsics	42	0	1	0	1
dict	X	X	X	X	X
hashing	X	X	X	X	X
iobuffer	91	1	1	0	4
staged	X	X	X	X	X
offsetarray	X	X	X	X	X
arrayops	X	X	X	X	X
tuple	0	0	0	0	3
reduce	102	4	12	0	39
reducedim	X	X	X	X	X
abstractarray	X	X	X	X	X
intfuncs	X	X	X	X	X
simdloop	X	X	X	X	X
vecelement	X	X	X	X	X
rational	98633	0	1	1	13
bitarray	2968	0	1	0	20
copy	0	0	0	0	4
math	117538	0	0	0	31
fastmath	928	0	0	0	4
functional	23	0	0	0	0
iterators	5627	0	4147	0	12
operators	45	0	0	0	5
ordering	2	0	0	0	0
path	0	0	0	0	0
ccall	X	X	X	X	X
parse	X	X	X	X	X
loading	X	X	X	X	X
gmp	X	X	X	X	X
sorting	97	0	0	0	9
spawn	X	X	X	X	X
backtrace	X	X	X	X	X
exceptions	5	8	4	0	4
file	X	X	X	X	X
read	25	1	0	0	3
version	2452	0	0	0	9
namedtuple	205	0	2	0	3
mpfr	0	0	0	0	8
broadcast	247	0	2	0	26
complex	X	X	X	X	X
floatapprox	49	0	0	0	2
reflection	X	X	X	X	X
regex	0	0	0	0	0
float16	762091	0	0	0	2
combinatorics	170	0	0	0	3
sysinfo	4	0	0	0	1
env	1	0	0	0	5
rounding	112552	0	144	0	4
mod2pi	X	X	X	X	X
euler	X	X	X	X	X
show	X	X	X	X	X
client	0	4	0	0	2
errorshow	X	X	X	X	X
sets	X	X	X	X	X
goto	19	0	0	0	0
llvmcall	X	X	X	X	X
llvmcall2	7	0	0	0	1
ryu	31194	0	21	0	1
some	71	0	0	0	0
meta	57	2	3	0	5
stacktraces	X	X	X	X	X
docs	X	X	X	X	X
misc	X	X	X	X	X
threads	X	X	X	X	X
binaryplatforms	4	0	46	0	8
atexit	0	0	0	0	0
enums	99	0	0	0	5
cmdlineargs	X	X	X	X	X
int	X	X	X	X	X
interpreter	3	0	0	0	3
checked	X	X	X	X	X
bitset	195	0	0	0	11
floatfuncs	202	0	0	0	2
boundscheck	X	X	X	X	X
error	0	0	0	0	0
cartesian	135	0	3	0	5
osutils	57	0	0	0	2
channels	X	X	X	X	X
iostream	0	0	0	0	1
secretbuffer	0	0	0	0	0
specificity	X	X	X	X	X
reinterpretarray	X	X	X	X	X
syntax	X	X	X	X	X
corelogging	0	0	0	0	0
missing	41	0	0	1	9
asyncmap	304	2	2	0	9
smallarrayshrink	36	0	0	0	0
opaque_closure	X	X	X	X	X
filesystem	0	0	0	0	0
download	X	X	X	X	X
SparseArrays/higherorderfns	11	2	0	0	15
SparseArrays/sparse	502	0	0	0	123
SparseArrays/sparsevector	112	0	1	0	33
LinearAlgebra/triangular	X	X	X	X	X
LinearAlgebra/qr	489	0	0	0	20
LinearAlgebra/dense	8571	0	24	0	35
LinearAlgebra/matmul	X	X	X	X	X
LinearAlgebra/schur	11	0	0	0	4
LinearAlgebra/special	2514	0	0	0	7
LinearAlgebra/eigen	34	0	1	0	8
LinearAlgebra/bunchkaufman	5688	0	0	0	12
LinearAlgebra/svd	0	0	0	0	7
LinearAlgebra/lapack	798	3	0	0	36
LinearAlgebra/tridiag	21	0	0	0	12
LinearAlgebra/bidiag	695	0	0	0	10
LinearAlgebra/diagonal	2	0	0	0	21
LinearAlgebra/cholesky	143	0	3	0	13
LinearAlgebra/lu	1225	3	1	0	15
LinearAlgebra/symmetric	2605	0	6	0	21
LinearAlgebra/generic	517	1	2	0	20
LinearAlgebra/uniformscaling	X	X	X	X	X
LinearAlgebra/lq	2856	0	0	0	8
LinearAlgebra/hessenberg	349	0	11	17	2
LinearAlgebra/blas	30	0	2	0	4
LinearAlgebra/adjtrans	264	0	0	0	14
LinearAlgebra/pinv	292	0	0	0	1
LinearAlgebra/givens	7	0	0	0	0
LinearAlgebra/structuredbroadcast	645	0	0	0	6
LinearAlgebra/addmul	X	X	X	X	X
LinearAlgebra/ldlt	7	0	0	0	3
LinearAlgebra/factorization	6	0	0	0	1
LibGit2/libgit2	13	0	0	0	5
Dates/accessors	7723858	0	0	0	6
Dates/adjusters	X	X	X	X	X
Dates/query	X	X	X	X	X
Dates/periods	227	0	0	0	14
Dates/ranges	350537	0	1	0	10
Dates/rounding	X	X	X	X	X
Dates/types	231	0	0	0	3
Dates/io	222	0	0	0	19
Dates/arithmetic	376	0	1	0	4
Dates/conversions	160	0	0	0	2
ArgTools	2	5	6	0	3
Artifacts	0	0	0	0	3
Base64	1006	0	1001	0	4
CRC32c	X	X	X	X	X
CompilerSupportLibraries_jll	4	0	0	0	0
DelimitedFiles	X	X	X	X	X
Downloads	144	0	15	0	1
FileWatching	X	X	X	X	X
Future	0	0	0	0	0
GMP_jll	0	0	1	0	1
InteractiveUtils	X	X	X	X	X
LLVMLibUnwind_jll	0	0	0	0	0
LazyArtifacts	1	0	0	0	1
LibCURL	0	0	0	0	0
LibCURL_jll	1	0	0	0	0
LibGit2_jll	2	0	0	0	0
LibSSH2_jll	0	0	0	0	0
LibUV_jll	1	0	0	0	0
LibUnwind_jll	0	0	0	0	0
Libdl	69	0	0	0	1
Logging	8	0	1	0	1
MPFR_jll	1	0	0	0	0
Markdown	241	1	0	0	82
MbedTLS_jll	1	0	0	0	0
Mmap	134	2	0	0	63
MozillaCACerts_jll	1	0	0	0	0
NetworkOptions	2575	0	929	0	4
OpenBLAS_jll	1	0	0	0	0
OpenLibm_jll	1	0	0	0	0
PCRE2_jll	2	0	0	0	0
Printf	669	0	1	0	1
Profile	X	X	X	X	X
REPL	1250	0	0	4	0
Random	3701	0	0	0	44
SHA	X	X	X	X	X
Serialization	X	X	X	X	X
Sockets	X	X	X	X	X
Statistics	713	0	6	0	18
SuiteSparse	921	0	0	0	1
SuiteSparse_jll	1	0	0	0	0
TOML	372	0	43	8	0
Tar	78	0	40	0	13
Test	X	X	X	X	X
UUIDs	1029	0	0	0	20
Unicode	19	0	0	0	6
Zlib_jll	1	0	0	0	0
dSFMT_jll	1	0	0	0	0
libLLVM_jll	1	0	0	0	0
libblastrampoline_jll	1	0	0	0	0
nghttp2_jll	1	0	0	0	0
p7zip_jll	1	0	0	0	0

[goerch]

Lots of ranges tests fail due to

high precision: Error During Test at C:\Users\Win10\AppData\Local\Programs\Julia-1.7.0\share/julia/test\ranges.jl:87
  Test threw exception
  Expression: cmp_sn(widen(x) + widen(y), hi, lo)
  MethodError: no method matching cmp_sn(::Float32, ::Float16, ::Float16, ::Int64)
  Closest candidates are:
    cmp_sn(::Any, ::Any, ::Any) at C:\Users\Win10\AppData\Local\Programs\Julia-1.7.0\share/julia/test\ranges.jl:35
  Stacktrace:
   [1] cmp_sn(w::Float32, hi::Float16, lo::Float16)
     @ Main.JuliaTests C:\Users\Win10\AppData\Local\Programs\Julia-1.7.0\share\julia\test\ranges.jl:36
   [2] macro expansion
     @ C:\Users\Win10\AppData\Local\Programs\Julia-1.7.0\share\julia\stdlib\v1.7\Test\src\Test.jl:445 [inlined]
   [3] highprec_pair(x::Float16, y::Float16)
     @ Main.JuliaTests C:\Users\Win10\AppData\Local\Programs\Julia-1.7.0\share\julia\test\ranges.jl:87

[goerch]

Regarding failures in numbers: in the Julia test we find

    real_types = [Base.BitInteger64_types...,
                  [Rational{T} for T in Base.BitInteger64_types]...,
                  Float32, Float64]
    for A = real_types, B = real_types
        T = promote_type(A,B)
        @test typeof(Complex(convert(A,2),convert(B,3))) <: Complex{T}
    end

Putting it into a let block this succeeds...

[goerch]

Test distribution also looks broken on Windows. With 5 workers I see:

[goerch]

Learning by doing: I'm excluding stdlib for now and get with my latest commit

Julia Version 1.7.0
Commit 3bf9d17731 (2021-11-30 12:12 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-10710U CPU @ 1.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake)
Environment:
  JULIA_EDITOR = code
  JULIA_NUM_THREADS = 5
Test run at: 2021-12-12T00:11:31.538

Maximum number of statements per lowered expression: 100000

Test file	Passes	Fails	Errors	Broken	Aborted blocks
ambiguous	105	0	1	2	14
compiler/inference	997	5	2	2	3
compiler/validation	24	0	1	0	0
compiler/ssair	X	X	X	X	X
compiler/irpasses	X	X	X	X	X
compiler/codegen	X	X	X	X	X
compiler/inline	63	2	0	1	0
compiler/contextual	9	2	1	0	1
subarray	318276	0	1	0	5
strings/basic	87674	0	0	0	5
strings/search	692	0	0	0	1
strings/util	619	0	0	0	6
strings/io	12764	0	0	0	2
strings/types	2302691	0	0	0	4
unicode/utf8	19	0	0	0	0
core	X	X	X	X	X
atomics	3444	0	0	0	0
keywordargs	151	0	0	0	0
numbers	1577509	0	1	1	16
subtype	X	X	X	X	X
char	1625	0	0	0	3
triplequote	29	0	0	0	0
intrinsics	301	0	0	0	0
dict	X	X	X	X	X
hashing	12521	0	0	0	7
iobuffer	205	0	0	0	1
staged	58	6	0	0	0
offsetarray	X	X	X	X	X
arrayops	1996	2	5	2	24
tuple	606	0	0	0	0
reduce	8578	2	0	0	13
reducedim	865	0	0	0	14
abstractarray	55292	0	1	24795	8
intfuncs	215863	0	1	0	6
simdloop	X	X	X	X	X
vecelement	678	0	0	0	1
rational	98633	0	1	1	7
bitarray	381657	0	3661	0	23
copy	331	0	2	0	2
math	X	X	X	X	X
fastmath	946	0	0	0	1
functional	98	0	0	0	0
iterators	10080	0	0	0	4
operators	13039	0	0	0	2
ordering	35	0	0	0	0
path	1051	0	0	12	2
ccall	X	X	X	X	X
parse	16098	0	0	0	4
loading	153389	2	1	0	15
gmp	2323	1	0	0	6
sorting	12649	0	8	10	8
spawn	X	X	X	X	X
backtrace	8	11	17	1	2
exceptions	38	24	7	0	3
file	X	X	X	X	X
read	3733	0	0	0	3
version	2452	0	0	0	1
namedtuple	214	0	0	0	1
mpfr	1099	13	15	1	6
broadcast	507	1	0	0	6
complex	8432	0	0	5	7
floatapprox	49	0	0	0	0
reflection	X	X	X	X	X
regex	128	0	2	0	1
float16	762091	0	0	0	1
combinatorics	170	0	0	0	2
sysinfo	4	0	0	0	0
env	96	0	1	0	4
rounding	112720	0	0	0	3
ranges	12110676	5	26	327699	19
mod2pi	80	0	0	0	0
euler	12	0	0	0	6
show	X	X	X	X	X
client	0	4	0	0	1
errorshow	X	X	X	X	X
sets	X	X	X	X	X
goto	19	0	0	0	0
llvmcall	X	X	X	X	X
llvmcall2	7	0	0	0	0
ryu	31194	0	21	0	1
some	71	0	0	0	0
meta	66	2	1	0	0
stacktraces	X	X	X	X	X
docs	X	X	X	X	X
misc	X	X	X	X	X
threads	X	X	X	X	X
stress	0	0	0	0	0
binaryplatforms	334	0	7	0	7
atexit	40	0	0	0	1
enums	99	0	0	0	0
cmdlineargs	X	X	X	X	X
int	X	X	X	X	X
interpreter	3	0	0	0	0
checked	1239	0	0	0	1
bitset	195	0	0	0	3
floatfuncs	208	0	13	0	2
precompile	X	X	X	X	X
boundscheck	X	X	X	X	X
error	31	0	0	0	0
cartesian	236	0	1	3	3
osutils	57	0	0	0	0
channels	X	X	X	X	X
iostream	50	0	0	0	1
secretbuffer	X	X	X	X	X
specificity	X	X	X	X	X
reinterpretarray	232	0	0	0	3
syntax	X	X	X	X	X
corelogging	55	2	21	0	1
missing	563	0	2	1	3
asyncmap	X	X	X	X	X
smallarrayshrink	36	0	0	0	0
opaque_closure	X	X	X	X	X
filesystem	4	0	0	0	0
download	X	X	X	X	X

My problems with ranges and numbers have vanished.

[goerch]

For a completely different take on this problem see here. Having made that experience I'd like to propose the following changes:

• Maintaining a local copy of Julia tests in the JuliaInterpreter repository to possibly deactivate crashing tests and keep reproducible results
• Limiting the execution of Julia not by number of statements but by depth of call stack
• Maybe defining interpreter instances collecting the internal caches (if these are even needed?)

[goerch]

We can now somehow compare the results from here and there. It seems to me there is a bit room for improvement. Do you have any proposals how to best proceed?