flex -S: m4preproc in skeletons not processed

[jannick0]

It appears that flex does not process m4-preprocedures m4preproc_... in skeletons when using flex -S.

As a sideremark: flex 2.6.4 requires that the skeleton file must not contain any comment lines (leading tag %#), bug fixed by 7af066b.

Example - output of flex -S flex.skl:

#line 2 "lex.yy.c"

#define  YY_INT_ALIGNED short int

/* A lexical scanner generated by flex */

m4preproc_changecom

#define FLEX_SCANNER
#define YY_FLEX_MAJOR_VERSION FLEX_MAJOR_VERSION
#define YY_FLEX_MINOR_VERSION FLEX_MINOR_VERSION
#define YY_FLEX_SUBMINOR_VERSION FLEX_SUBMINOR_VERSION
#if YY_FLEX_SUBMINOR_VERSION > 0
#define FLEX_BETA
#endif

m4preproc_define(`M4_GEN_PREFIX',``
#ifdef yy$1
#define yy$1_ALREADY_DEFINED
#else
#define yy$1 yy$1
#endif
'm4preproc_divert(1)`
#ifndef yy$1_ALREADY_DEFINED
#undef yy$1
#endif'm4preproc_divert(0)')

/* First, we deal with  platform-specific or compiler-specific issues. */

/* begin standard C headers. */
#ifndef _POSIX_C_SOURCE
#define _POSIX_C_SOURCE 1 /* for fileno() */
#ifndef _POSIX_SOURCE
#define _POSIX_SOURCE 1
#endif
#endif
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <stdlib.h>

/* end standard C headers. */

m4preproc_include(`flexint.h')
...

[Explorer09]

I would argue that it's part of the design to not process m4preproc macros in Flex.

[jannick0]

I would rather consider this a bug if flex does not process the very skeleton which is compiled into flex.

[zmajeed]

I was not able to find any docs beyond info flex for the flex -S option:

'-SFILE, --skel=FILE'
     overrides the default skeleton file from which 'flex' constructs its scanners.
     You'll never need this option unless you are doing 'flex' maintenance or development.

The source flex.skl is actually embedded into the flex binary so there is no installed skeleton file to look at or modify

Looking at the source it seems flex.skl is preprocessed into a C file skel.c by mkskel.sh (Makefile.am). The C file looks like this:

/* skel.c */
/* File created from flex.skl via mkskel.sh */
#include "flexdef.h"
const char *skel[] = {
  "/* A lexical scanner generated by flex */",
/* more string entries comprising skeleton file */
  "#undef yyTABLES_NAME",
  "#endif]]",
  "]])",
  0
};

The strings in the skel array are combined into the final skeleton file by skelout in misc.c

This allowed me to generate a skeleton file from flex.skl that works for flex -S. I'll post the script for it in a following comment

[zmajeed]

A script skelfromc.sh to generate a skeleton file from flex.skl that works with flex -S. See above comment for explanation.

#!/bin/bash

# skelfromc.sh

# a script to generate a skeleton file that works for the flex -S option
# extracts and combines strings from the C code output by mkskel.sh

# usage:
# mkskel.sh flex/src m4 2.6.4 | skelfromc.sh >custom.skl
# flex -S custom.skl rules.l

# delete lines not starting with doublequote with optional leading spaces
# strip initial and terminal doublequotes and trailing comma
# unescape backslashes
# unescape embedded doublequotes

sed '
/^ *"/!d
s///
s/", *$//
s/\\\\/\\/g
s/\\"/"/g
' 

Run as

$ mkskel.sh flex/src m4 2.6.4 | skelfromc.sh >custom.skl
$ flex -S custom.skl rules.l