comment brittleness

[nikodemus]

Current parser design (including self-hosted one) is brittle and buggy vs comments: since comments are parsed as distinct tokens they need to be taken into account separately in all non-expression locations they can appear:

import -- this needs special handling
   bar.*

class -- this needs special handling
   Foo
   -- this needs special handling
   {}
end

let -- this needs special handling
  x = 42.

[ 1, -- this needs special handling
  2 ]

...etc

Plan

Change the parser so that comments become properties of other tokens: ie. make "---Foo--- Bar" tokenize as #<TokenWord Bar prefixBlockComment: "Foo"> regardless of if it is in expression or non-expression context.

Relates issues

• #590
• #487