D skeleton file breaks recent D compilers with example code from manual

[ledaniel2]

The following code uses a definition of reportSyntaxError taken from the bison manual $10.2.6. The result compiles with gdc version 10.2.1 but not with more recent versions of gdc, ldc and dmd.exe. (Note the change to lalr1.d described in issue #84 is needed, too. The version of bison tested was 3.8.2.12-013d.)

%language "D"
%output "bison_bug.d"
%define parse.error custom
%locations
%define api.value.type union

%token <int> DUMMY SYNTAXERROR 

%code lexer {

import std.stdio : stderr, writeln;

public void reportSyntaxError(YYParser.Context ctx)
{
  stderr.write(ctx.getLocation(), ": syntax error");
  // Report the expected tokens.
  {
    immutable int TOKENMAX = 5;
    YYParser.SymbolKind[] arg = new YYParser.SymbolKind[TOKENMAX];
    int n = ctx.getExpectedTokens(arg, TOKENMAX);
    if (n < TOKENMAX)
      for (int i = 0; i < n; ++i)
        stderr.write((i == 0 ? ": expected " : " or "), arg[i]);
  }
  // Report the unexpected token which triggered the error.
  {
    YYParser.SymbolKind lookahead = ctx.getToken();
    stderr.writeln(" before ", lookahead);
  }
}

public void yyerror(const(YYLocation) loc, string s) {
	stderr.writeln("error: ", s);
}

public Symbol yylex() {
	return Symbol(TokenKind.SYNTAXERROR, YYLocation());
}

}

%%

Stmts   : %empty
        | Stmts Stmt
        ;

Stmt	: DUMMY
	;

%%

int main() {
	auto parser = new YYParser();
	parser.parse();
	
	return 0;
}

Below is the output when compiling the resulting bison_bug.d with dmd.exe v2.098.1:

C:\Program Files\Digital Mars\dmd2\windows\bin\..\..\src\phobos\std\format\internal\write.d(166): Error: no property `yycode_` for type `bison_bug.YYParser.SymbolKind`
C:\Program Files\Digital Mars\dmd2\windows\bin\..\..\src\phobos\std\format\internal\write.d(248): Error: no property `yycode_` for type `bison_bug.YYParser.SymbolKind`
C:\Program Files\Digital Mars\dmd2\windows\bin\..\..\src\phobos\std\format\internal\write.d(248): Error: incompatible types for `(obj) != (0)`: `SymbolKind` and `int`
C:\Program Files\Digital Mars\dmd2\windows\bin\..\..\src\phobos\std\format\internal\write.d(253): Error: no property `yycode_` for type `bison_bug.YYParser.SymbolKind`
C:\Program Files\Digital Mars\dmd2\windows\bin\..\..\src\phobos\std\format\internal\write.d(253): Error: incompatible types for `(obj) != (0)`: `SymbolKind` and `int`
C:\Program Files\Digital Mars\dmd2\windows\bin\..\..\src\phobos\std\format\write.d(1239): Error: template instance `std.format.internal.write.formatValueImpl!(LockingTextWriter, SymbolKind, char)` error instantiating
C:\Program Files\Digital Mars\dmd2\windows\bin\..\..\src\phobos\std\format\write.d(632):        instantiated from here: `formatValue!(LockingTextWriter, SymbolKind, char)`
C:\Program Files\Digital Mars\dmd2\windows\bin\..\..\src\phobos\std\stdio.d(1719):        instantiated from here: `formattedWrite!(LockingTextWriter, char, SymbolKind)`
bison_bug.y(23):        instantiated from here: `write!(string, SymbolKind)`
C:\Program Files\Digital Mars\dmd2\windows\bin\..\..\src\phobos\std\stdio.d(1765): Error: template instance `std.stdio.File.write!(string, SymbolKind, char)` error instantiating
bison_bug.y(28):        instantiated from here: `writeln!(string, SymbolKind)`

The proposed fix is to apply the following changes to source file data/skeletons/d.m4, or the installed /usr/share/bison/skeletons/d.m4:

--- [...]/data/skeletons/d.m4
+++ /usr/share/bison/skeletons/d.m4
@@ -282,23 +282,22 @@
     /* Return YYSTR after stripping away unnecessary quotes and
      backslashes, so that it's suitable for yyerror.  The heuristic is
      that double-quoting is unnecessary unless the string contains an
      apostrophe, a comma, or backslash (other than backslash-backslash).
      YYSTR is taken from yytname.  */
-    final void toString(W)(W sink) const
-    if (isOutputRange!(W, char))
+    final void toString(void delegate(const(char)[]) sink) const
     {
       immutable string[] yy_sname = @{
   ]b4_symbol_names[
       @};]b4_has_translations_if([[
       /* YYTRANSLATABLE[SYMBOL-NUM] -- Whether YY_SNAME[SYMBOL-NUM] is
         internationalizable.  */
       immutable ]b4_int_type_for([b4_translatable])[[] yytranslatable = @{
   ]b4_translatable[
       @};]])[
 
-      put(sink, yy_sname[yycode_]);
+      sink.formattedWrite!"%s"(yy_sname[yycode_]);
     }
   }
 ]])

Now the output is as expected:

1.1: syntax error: expected end of file or DUMMY before SYNTAXERROR

A delegate is D's term for a lambda function which captures its immediately enclosing scope, and would appear to be the more "modern" way of implementing this pattern (see [Çehreli,2017] $72.4, p485). It should be valid D code for any compiler which can otherwise handle the D output of bison (implements static foreach etc.)

[Çehreli,2017] Programming in D, First Edition, Ali Çehreli, 2009-17

[akimd]

@adelavais, could you please review this proposal? Thanks in advance!

[adelavais]

I will review it towards the end of the week. Thanks for letting me know!

[akimd]

Btw, I would like to be able to state in the documentation what version of D we are supporting. I don't know how D "versions" are labeled. I don't know if there are "releases", as for Java, or standards, as for C++.

[cpp-tutor]

I don't really have an answer for supported version, other than I believe DMD is the "reference implementation", and the latest release version of this compiler is what we should aim to support? (Also, it was a coincidence I discovered that the existing skeleton works with gdc 10.x. As previously stated the proposed change does not, I believe, break older compiler versions which would otherwise compile ok - possibly this could be an area to look at in more detail.)

[akimd]

@adelavais,

There are more issues to check: please see also https://github.com/akimd/bison/issues/84.

Cheers!

[akimd]

@ledaniel2

Thanks for the report. You may check the fix in the following tarballs.

• https://www.lrde.epita.fr/~akim/private/bison/bison-3.8.2.44-de30.tar.gz 5.6M
• https://www.lrde.epita.fr/~akim/private/bison/bison-3.8.2.44-de30.tar.lz 3.1M
• https://www.lrde.epita.fr/~akim/private/bison/bison-3.8.2.44-de30.tar.xz 3.1M

I would need your email to be able to list you in THANKS.

Cheers!