Failures parsing General registry

[c42f]

Here's a list of failures when parsing General restricted to cases where

• JuliaSyntax.jl fails to parse a file
• the reference parser accepts the file
• or both parsers succeed, but the resulting Exprs mismatch

Parser failures

• [x] A[x ? y : end] - AssociativeArrays_0.0.3/src/namedaxis.jl, Reduce_1.2.12/src/Reduce.jl, UnROOT_0.8.1/src/UnROOT.jl
• [x] function (f()) where T end - DMRJtensor_0.8.7/lib/Qtensor.jl, DMRJtensor_0.8.7/lib/decompositions.jl, EnglishText_0.6.0/src/numeric.jl, ExprTools_0.1.8/test/function.jl, MemoryMutate_0.0.4/src/MemoryMutate.jl, MultivariatePolynomials_0.4.2/src/differentiation.jl, Singular_0.9.1/src/module/vector.jl, etc why people why??
• [x] .1..0.1 - DomainSets_0.5.9/test/test_setoperations.jl, 0x01..0xcf - Rasters_0.2.0/test/sources/gdal.jl
• [x] import Base.:* - Empirikos_0.4.5/src/ebayes_targets.jl, import A.:⋅ - MeasureTheory_0.13.2/test/runtests.jl, Pathnames_0.1.0/src/Pathnames.jl, Porta_0.1.4/src/geometry/biquaternions.jl
• [x] Base.(:*) - UnitfulLatexify_1.5.2/src/UnitfulLatexify.jl
• [x] function ($M).$J() end - Knet_1.4.9/src/cuarrays/reduction.jl
• [x] @a@b - MathematicalSystems_0.11.11/test/@system.jl
• [x] [a~b]- ModelingToolkitStandardLibrary_0.1.0/test/test_continuous.jl, NeuralPDE_4.4.0/test/NNPDE_tests.jl, etc
• [x] @(A) - NumericalMethodsforEngineers_2.0.0/examples/ch07/WRM/Fig.7.14.jl
• [x] -1.#comment - POMDPModels_0.4.15/src/MountainCar.jl
• [x] A.@a(x).b - WorkerUtilities_1.0.1/test/runtests.jl
• [x] (1+2).@x - julia_1.6.1/test/show.jl

Parses, but Expr is wrong

• [x] @.r - no space after macro name
• [x] @\ _ - symbol macro names
• [x] "[1 2\n]" - hvcat rather than hcat
• [x] A where {X, Y; Z} - parameters not in first args slot
• [x] @a(x; a) - parameters not in first args slot
• [x] try a end should be an error
• [x] function (f::T{g(i)})() end and function (::g(x))() end
• [x] .&(x)
• [x] +0xff
• [x] f(x for a = as, b = bs for c = cs, d = ds)
• [x] x ≂ y ≂ z
• [x] A[x, y; z = 1] - parameter block = vs kw
• [x] "\"\"\"\n\$x\n b\"\"\"" - wrong deindentation
• [x] using A: @. - not lowered to @__dot__
• [x] A{x;y,z} and A{x;y,z=1} - parameter blocks & kw vs =
• [x] function (:*=(f))() end - NiLang uses this

Bizarre forms which parse differently but unclear if it's worth fixing

• [ ] function (begin end) nothing end ExprTools_0.1.8/test/function.jl

Expr conversion failures

• [x] JuliaFormatter_0.22.0/test/files/PProf.jl
• [x] MathOptInterfaceMosek_0.5.2/examples/chainsing.jl
• [x] MosekTools_0.11.0/examples/chainsing.jl
• [x] SDFReader_0.3.4/test/0002.jl

Bugs in reference parser

• [ ] export @A @B should not parse - TSML_2.7.2/src/TSML.jl
• [ ] 17.328679084777833f0 is not correctly rounded?
• [ ] 1.0e+ parses as an Int128 literal
• [ ] f()isa(x,y) is not the juxtaposed multiplication f()*isa(x,y). It's infix isa syntax instead.
• [ ] [i; for i in is] is allowed by the reference parser but the presence of this semicolon is weird and broken.

The following bugs we can deal with when comparing Exprs

• [x] 0x1.8p23f is a Float64 literal, with the trailing f ignored (also 0x1p1f0)
• [x] The macrocall in "@f(a=1) do\nend" is not the same as the call in @f(a=1)
• [x] global (x,y) is the same as global x,y in reference parser
• [x] "\"\"\"\n a\n \n b\"\"\"" parses to "a\n \nb" in reference parser - lines with partial whitespace are handled inconsistently
• [x] (a; b,) - should be a frankentuple not a block. Occurs a single time in QuadraticFormsMGHyp_0.5.1/src/QuadraticFormsMGHyp.jl and appears to be a syntax mistake.
• [ ] "\"\"\"\n \$y\n \n \"\"\"\n" - partial whitespace
• [ ] "f(g() = x)" - extra block

Intentional failures due to being stricter than reference parser

• [x] import A as (a.b) - julia_1.6.1/test/syntax.jl
• [ ] Juxtaposition of strings with more keywords is disallowed. For example "str"for i=is end

[c42f]

Couple of issues which might be bugs in the reference parser:

• global (x,y)
• "@f(a=1) do\nend" parsed differently from @f(a=1)
• "\"\"\"\n a\n \n b\"\"\"" strips the mismatched whitespace weirdly

[oscardssmith]

Related to the Float64 literal bug you found, 0x1p1f0 is also valid (but 0x1p1f1 isn't)

[c42f]

Thanks yes the float literal bug is pretty weird. I tried to understand how this happens a while back. IIRC it was a combination of (a) a bug in the reference lexer allowing extra sufficies and (b) reusing the flisp reader for numeric literals which does allow f suffix to parse Float32 literals. Something like that... it was subtle.

We had a call about the issues mentioned in https://github.com/JuliaLang/JuliaSyntax.jl/issues/134#issuecomment-1331552385 and decided they're bugs. It's a little unclear what to do about the string processing issue. But clear that the existing behavior is weird.

[c42f]

@oscardssmith I've found a weird disagreement when parsing the float constant 17.328679084777833f0, thought you might be interested :)

I think the following indicate this literal gets incorrectly rounded in the reference parser:

julia> abs(big"17.328679084777833" - JuliaSyntax.parse(Expr, "17.328679084777833f0")) <
       abs(big"17.328679084777833" - JuliaSyntax.fl_parse("17.328679084777833f0"))
true

julia> x = JuliaSyntax.fl_parse("17.328679084777833f0")
17.328678f0

# looks like the error is roughly 1/2 ulps of a Float64
julia> abs(big"17.328679084777833" - x) - abs(big"17.328679084777833" - nextfloat(x))
1.937500000000000000000000000000000000000000000000000000000000163954795499142396e-15

[oscardssmith]

what's fl_parse?

[c42f]

fl_parse is just a direct call to the reference parser. It's useful when developing this library to use instead of Meta.parse(), as Meta.parse() may have been overridden with the newer parser.

[oscardssmith]

this isn't great. can you make a Julialang issue to track it?

[c42f]

I chatted to @JeffBezanson about a couple more of these issues today.

We thought that

• The current behavior of JuliaSyntax handling of edge cases in triple quoted string deindentation is fine and is the simplest rule. That is, only empty lines are special and ignored for the purposes of common prefix detection. If a line has nonzero whitespace it's treated like any other line.
• The text (a; b,) should indeed be a frankentuple not a block. We'll call that a minor bug in the reference parser.