sysprog21
Major frontend refactor
This patch aims to resolve most current preprocessor issues seen in parser unit, including:
- Inappropriate eager file inclusion expansion, this is caused due to file inclusion is always happening before actual if-else file guard can determine, which sometimes causes weird result.
- Bad error message reporting system, current system uses source index to suggest user where the error is, which is not really helpful on pratical term.
- Lack of expansion only compilation.
- Lack of predefined macros, including
__LINE__,__FILE__.
In this approach, we introduce a whole new phase dedicated to preprocessing (this practice can be seen in chibicc and other similar cc), instead of binding the preprocessing phase in few different places.
Current status
This draft is still in development, the below screenshot is comparison between result of cpp -E and out/shecc -E:
comparison
test.c:
#define B
#define EXP(...) __VA_ARGS__
#include "a.c"
int main()
{
#if defined(B)
__LINE__;
__FILE__;
#endif
EXP(1, +, 2);
printf("Hello, World!\n");
return 0;
}
a.c:
#pragma once
int fib(void)
{
A;
__FILE__;
return 0;
}
Current major compilation workflow hasn't affected by the patch, which is expected to resolve in later commits.
Memory overhead
Memory usage increasement is expected since we now use token_t struct to track position, and token stream will be generated at once for preprocessor and parser to use with. Once parser finished, we expect to free all tokens.
Summary by cubic
Introduces a dedicated preprocessing phase and rebuilds token handling to fix include guard behavior, add robust macro expansion, and improve error diagnostics. Adds an -E mode to output preprocessed code comparable to cpp -E.
-
New Features
- Standalone preprocessor with object/function-like macros, variadics, and hide sets.
- Conditional directives: #if, #ifdef, #ifndef, #elif, #else, #endif.
- #pragma once and proper include path parsing for "<...>" and "..." files.
- Predefined macros: LINE and FILE.
- -E expansion-only mode with preprocessed output emitter.
- Better token tracking and caching, with precise file:line:column locations.
- CI job to preprocess and build a stage1 artifact to validate -E output.
- libc: add fseek/ftell with lseek support for ARM/RISC-V.
-
Bug Fixes
- Include expansion now respects guards; files are included after condition evaluation.
- Error messages upgraded from raw source indices to readable diagnostics with caret highlights.
- Corrected #elif evaluation during preprocessing.
- Deferred string/char literal unescaping until after preprocessing; fixes hex and octal escape handling.
Written for commit 8d4198da58667a2bd106189cde2c2af57551718b. Summary will update automatically on new commits.
The initial result is benchmarked below, the memory overhead and minor performance overhead are expected to be the outcome of algorithm and data structure used in this patch:
- The source files will be loaded once into memory (which used for later error diagnositc and tokenization)
- The token stream will later be computed (or tokenized) and store in
TOKEN_ARENA, and this is the major memory overhead expected to be produced in this patch - The computed token stream of each source files will also get cached
- Later in preprocessor, each token will also get copied to be preprocessed, this is to prevent corrupted token integrity if later cached token stream is used. This is also expected to produce another major memory overhead.
- After the preprocessing is done, comipler will pass the whole computed token stream to parser
The performance overhead is expected to be came from multiuple token stream traversal, as seen 3 times at least in lexer, preprocessor, and parser.
performance benchmark (stage 0)
Before
/bin/time -v out/shecc src/main.c:
Command being timed: "./out/shecc ./src/main.c"
User time (seconds): 0.16
System time (seconds): 0.33
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.50
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 305664
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 76195
Voluntary context switches: 0
Involuntary context switches: 2
Swaps: 0
File system inputs: 0
File system outputs: 728
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
After
/bin/time -v out/shecc src/main.c:
Command being timed: "./out/shecc ./src/main.c"
User time (seconds): 0.20
System time (seconds): 0.44
Percent of CPU this job got: 98%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.66
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 384000
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 3
Minor (reclaiming a frame) page faults: 96004
Voluntary context switches: 44
Involuntary context switches: 2
Swaps: 0
File system inputs: 1992
File system outputs: 800
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
performance benchmark (stage 1)
Before
/bin/time -v out/shecc-stage1.elf src/main.c:
Command being timed: "./out/shecc-stage1.elf ./src/main.c"
User time (seconds): 1.49
System time (seconds): 1.23
Percent of CPU this job got: 98%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.76
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 188772
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 64
Minor (reclaiming a frame) page faults: 44474
Voluntary context switches: 158
Involuntary context switches: 5
Swaps: 0
File system inputs: 10880
File system outputs: 728
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
After
/bin/time -v out/shecc-stage1.elf src/main.c:
Command being timed: "./out/shecc-stage1.elf ./src/main.c"
User time (seconds): 1.94
System time (seconds): 1.50
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:03.47
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 233612
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 63
Minor (reclaiming a frame) page faults: 55761
Voluntary context switches: 50
Involuntary context switches: 15
Swaps: 0
File system inputs: 9120
File system outputs: 800
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
There are still some improvements left to do, e.g.:
- Delay string / character literal escaped character computation, so that when passing
-Ethe result don't have to convert back again. - Function & structure renaming, some leftover unused structures will be also removed as well.
I seem to unable to reply to the latest review suggestion so I'll reply here:
Would it be better to use a global array to store the strings and use
token->kindas the index to retrieve the corresponding one?
Perhaps? The approach requires reorganization of token kind order, and I don't think it's necessary to just boost performance while debugging (I didn't encoutered significant performance overhead when calling it)?
I'm thinking an approach to validate our compiler's expansion only compilation flag (aka -E), the procedure is as follow:
make distclean config out/shecc./out/shecc -E src/main.c > out/out.c./out/shecc --no-libc -o out/shecc-stage1.elf out/out.c- If the above procedure succeeded, we may continue to bootstrap stage 2 executable with the above procedure as well
@jserv what do you think?
I'm thinking an approach to validate our compiler's expansion only compilation flag (aka
-E), the procedure is as follow:
make distclean config out/shecc./out/shecc -E src/main.c > out/out.c
Furthermore, we can adapt the preprocessor implementation from slimcc.
#221 is related to this patch, as this patch also fixes error diagnostic context issue. We may also close it upon merged into master branch.
It seems all static test stuite failed on stage 2 compilation after merge conflicts resolution and minor change introduced, which is weird.
After adding the -strace flag, the output shows that lseek() reports a "Bad file descriptor" error.
$ qemu-arm -strace out/shecc-stage1.elf --no-libc -o test test.c
...
11316 lseek(-9,0,SEEK_CUR) = -1 errno=9 (Bad file descriptor)
11316 lseek(-9,0,SEEK_CUR) = -1 errno=9 (Bad file descriptor)
11316 lseek(-9,0,SEEK_CUR) = -1 errno=9 (Bad file descriptor)
11316 lseek(-9,0,SEEK_CUR) = -1 errno=9 (Bad file descriptor)
11316 lseek(-9,0,SEEK_CUR) = -1 errno=9 (Bad file descriptor)
11316 lseek(-9,0,SEEK_CUR) = -1 errno=9 (Bad file descriptor)
