cplusplus
cplusplus copied to clipboard
Published
20 hours ago •
robertoraggi
robertoraggi
Memory usage and permissive parsing with sqlite3
Trying to parse sqlite3.c with this project I found that it can parse it without showing any error/warnings but uses a bit more memory than gcc/g++ (cxx=3,000,000 allocations, g++=500,000 allocations):
/usr/bin/time ./cxx -toolchain linux ./sqlite3.cpp -fsyntax-only -I.
1.01user 0.06system 0:01.08elapsed 99%CPU (0avgtext+0avgdata 146628maxresident)k
0inputs+0outputs (0major+39686minor)pagefaults 0swaps
/usr/bin/time g++ -fpermissive ./sqlite3.cpp -fsyntax-only -I.
1.08user 0.07system 0:01.16elapsed 99%CPU (0avgtext+0avgdata 115564maxresident)k
0inputs+0outputs (0major+29614minor)pagefaults 0swaps
/usr/bin/time myvalgrind ./cxx -toolchain linux ./sqlite3.cpp -fsyntax-only -I.
==21479== in use at exit: 0 bytes in 0 blocks
==21479== total heap usage: 3,003,109 allocs, 3,003,109 frees, 4,175,102,281 bytes allocated
55.95user 0.31system 0:56.27elapsed 99%CPU (0avgtext+0avgdata 408920maxresident)k
0inputs+0outputs (0major+127697minor)pagefaults 0swaps
/usr/bin/time myvalgrind --trace-children=yes g++ -fpermissive ./sqlite3.cpp -fsyntax-only -I.
==21623== Command: g++ -fpermissive ./sqlite3.cpp -fsyntax-only -I.
==21624== in use at exit: 7,045,865 bytes in 15,051 blocks
==21624== total heap usage: 490,444 allocs, 475,393 frees, 649,209,160 bytes allocated
==21624== Command: /usr/lib/gcc/x86_64-linux-gnu/9/cc1plus -quiet -I . -imultiarch x86_64-linux-gnu -D_GNU_SOURCE ./sqlite3.cpp -quiet -dumpbase sqlite3.cpp -mtune=generic -march=x86-64 -auxbase sqlite3 -fpermissive -fsyntax-only -o /dev/null -fasynchronous-unwind-tables -fstack-protector-strong -Wformat -Wformat-security
==21623== in use at exit: 189,449 bytes in 69 blocks
==21623== total heap usage: 234 allocs, 165 frees, 215,031 bytes allocated
40.27user 0.34system 0:40.63elapsed 99%CPU (0avgtext+0avgdata 337688maxresident)k
4416inputs+0outputs (1major+147633minor)pagefaults 0swaps
But renaming sqlite3.c to sqlite3.cpp and fixing several errors pointed out by g++ then this project still doesn't show any warning/error while g++ needs -fpermissive to parse it and show several warnings:
./sqlite3.cpp: In function ‘void* sqlite3MemMalloc(int)’:
./sqlite3.cpp:26707:44: warning: invalid conversion from ‘void*’ to ‘sqlite3_int64*’ {aka ‘long long int*’} [-fpermissive]
26707 | #define SQLITE_MALLOC(x) malloc(x)
| ~~~~~~^~~
| |
| void*
./sqlite3.cpp:26773:7: note: in expansion of macro ‘SQLITE_MALLOC’
26773 | p = SQLITE_MALLOC( nByte+8 );
| ^~~~~~~~~~~~~
./sqlite3.cpp: In function ‘void* sqlite3MemRealloc(void*, int)’:
./sqlite3.cpp:26709:45: warning: invalid conversion from ‘void*’ to ‘sqlite3_int64*’ {aka ‘long long int*’} [-fpermissive]
26709 | #define SQLITE_REALLOC(x,y) realloc((x),(y))
| ~~~~~~~^~~~~~~~~
| |
| void*
...
./sqlite3.cpp: In function ‘int jsonEachNext(sqlite3_vtab_cursor*)’:
./sqlite3.cpp:209852:32: warning: invalid conversion from ‘void*’ to ‘JsonParent*’ [-fpermissive]
209852 | pNew = sqlite3DbRealloc(p->db, p->aParent, sizeof(JsonParent)*nNew);
| ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| |
| void*
./sqlite3.cpp: In function ‘int jsonEachFilter(sqlite3_vtab_cursor*, int, const char*, int, sqlite3_value**)’:
./sqlite3.cpp:210179:37: warning: invalid conversion from ‘void*’ to ‘JsonParent*’ [-fpermissive]
210179 | p->aParent = sqlite3DbMallocZero(p->db, sizeof(JsonParent));
| ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~
| |
| void*
I'm attaching the files used by this test: sqlite3-c-cpp.zip
Here is the output of valgrind with clang-17 (1,000,000 allocations):
/usr/bin/time myvalgrind --trace-children=yes clang-17-env clang ./sqlite3.c -c -I.
==18426== Memcheck, a memory error detector
==18426== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==18426== Using Valgrind-3.21.0 and LibVEX; rerun with -h for copyright info
==18426== Command: /home/mingo/bin/clang-17-env clang ./sqlite3.c -c -I.
==18426==
==18427== Memcheck, a memory error detector
==18427== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==18427== Using Valgrind-3.21.0 and LibVEX; rerun with -h for copyright info
==18427== Command: /home/mingo/bin/clang-3-env-base clang ./sqlite3.c -c -I.
==18427==
==18428== Memcheck, a memory error detector
==18428== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==18428== Using Valgrind-3.21.0 and LibVEX; rerun with -h for copyright info
==18428== Command: /home/mingo/local/clang-17/bin/clang ./sqlite3.c -c -I.
==18428==
==18428==
==18428== HEAP SUMMARY:
==18428== in use at exit: 44,223,475 bytes in 324,547 blocks
==18428== total heap usage: 978,232 allocs, 653,685 frees, 536,676,783 bytes allocated
==18428==
==18428== LEAK SUMMARY:
==18428== definitely lost: 0 bytes in 0 blocks
==18428== indirectly lost: 0 bytes in 0 blocks
==18428== possibly lost: 34,882,928 bytes in 301,772 blocks
==18428== still reachable: 9,340,547 bytes in 22,775 blocks
==18428== suppressed: 0 bytes in 0 blocks
==18428== Rerun with --leak-check=full to see details of leaked memory
==18428==
==18428== For lists of detected and suppressed errors, rerun with: -s
==18428== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==18427==
==18427== HEAP SUMMARY:
==18427== in use at exit: 48,333 bytes in 775 blocks
==18427== total heap usage: 1,491 allocs, 716 frees, 83,250 bytes allocated
==18427==
==18427== LEAK SUMMARY:
==18427== definitely lost: 281 bytes in 1 blocks
==18427== indirectly lost: 0 bytes in 0 blocks
==18427== possibly lost: 0 bytes in 0 blocks
==18427== still reachable: 48,052 bytes in 774 blocks
==18427== suppressed: 0 bytes in 0 blocks
==18427== Rerun with --leak-check=full to see details of leaked memory
==18427==
==18427== For lists of detected and suppressed errors, rerun with: -s
==18427== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==18426==
==18426== HEAP SUMMARY:
==18426== in use at exit: 44,798 bytes in 692 blocks
==18426== total heap usage: 974 allocs, 282 frees, 64,546 bytes allocated
==18426==
==18426== LEAK SUMMARY:
==18426== definitely lost: 281 bytes in 1 blocks
==18426== indirectly lost: 0 bytes in 0 blocks
==18426== possibly lost: 0 bytes in 0 blocks
==18426== still reachable: 44,517 bytes in 691 blocks
==18426== suppressed: 0 bytes in 0 blocks
==18426== Rerun with --leak-check=full to see details of leaked memory
==18426==
==18426== For lists of detected and suppressed errors, rerun with: -s
==18426== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
96.97user 0.55system 1:37.67elapsed 99%CPU (0avgtext+0avgdata 507316maxresident)k
85352inputs+3016outputs (2major+181810minor)pagefaults 0swaps
Notice that I like that this project can parse C source files but when they are renamed from .c to .cpp I would expect it to be a bit more C++ strict.
@mingodad Most memory is used for temporary tokens and data structures during preprocessing, along with symbols in the scope chain. The AST is stored in an arena, which should be fine. I'm working on code to reduce the preprocessor memory usage, then we should see numbers similar to other C++ frontends.
Thank you for reply !
Attached is the dirty changes I made to instrument the code to show timing and allocation while parsing sqlite3.c:
fprintf(stderr, "%s: Time taken %d seconds %d milliseconds, %'zd : %'zd : %'zd : %'zd : %'zd : %'zd\n", title, msec/1000, msec%1000, getCurrentRSS(), getPeakRSS(),
malloc_count, free_count, malloc_count-malloc_prev_count, free_count-free_prev_count);
/usr/bin/time ./cxx -toolchain linux ./sqlite3.c -c -I.
start main: Time taken 0 seconds 1 milliseconds, 2150400 : 2150400 : 247 : 38 : 247 : 38
before readAll: Time taken 0 seconds 1 milliseconds, 4812800 : 4812800 : 4970 : 3211 : 4723 : 3173
after unit.setSource: Time taken 0 seconds 817 milliseconds, 146423808 : 146423808 : 188992 : 144030 : 184022 : 140819
before preprocesor->squeeze: Time taken 0 seconds 0 milliseconds, 137555968 : 146423808 : 188992 : 144031 : 0 : 1
before unit.parse: Time taken 0 seconds 1 milliseconds, 137555968 : 146423808 : 188992 : 150101 : 0 : 6070
declaration: Time taken 0 seconds 0 milliseconds, 137555968 : 146423808 : 189020 : 150101 : 28 : 0
/usr/lib/gcc/x86_64-linux-gnu/11/include/stdarg.h:40:1: 1 : 2 : 1266
declaration: Time taken 0 seconds 0 milliseconds, 137555968 : 146423808 : 189026 : 150104 : 6 : 3
/usr/lib/gcc/x86_64-linux-gnu/11/include/stdarg.h:99:1: 2 : 2 : 3405
declaration: Time taken 0 seconds 0 milliseconds, 137555968 : 146423808 : 189038 : 150106 : 12 : 2
./sqlite3.c:500:12: 3 : 2 : 19459
declaration: Time taken 0 seconds 0 milliseconds, 137555968 : 146423808 : 189056 : 150108 : 18 : 2
./sqlite3.c:501:12: 4 : 2 : 19517
...
declaration: Time taken 0 seconds 0 milliseconds, 150380544 : 150380544 : 3002651 : 2453607 : 23 : 15
./sqlite3.c:257674:12: 4589 : 11 : 9088707
after unit.parse: Time taken 0 seconds 0 milliseconds, 150380544 : 150380544 : 3002651 : 2455111 : 0 : 1504
before diagnosticsClient.verifyExpectedDiagnostics: Time taken 0 seconds 0 milliseconds, 150380544 : 150380544 : 3002651 : 2455111 : 0 : 0
end main: Time taken 0 seconds 44 milliseconds, 129077248 : 150380544 : 3002651 : 3002439 : 0 : 547328
Total tokens = 766945
Total declarations = 4589
1.73user 0.17system 0:01.93elapsed 98%CPU (0avgtext+0avgdata 146856maxresident)k
0inputs+0outputs (0major+41904minor)pagefaults 0swaps
Doing the above experiment I've noticed that using the code shown bellow to show where the declaration start doesn't work in all cases because the Token is reused and only register the initial location, in sqlite3.c most of the functions start with SQLITE_PRIVATE and its definition is at the top of the file.
const Token& tk = unit->tokenAt(declaration->firstSourceLocation());
unsigned line, column;
std::string_view fileName;
unit->preprocessor()->getTokenStartPosition(tk, &line, &column, &fileName);
printf("%.*s:%u:%u: %zd : %d : %u\n", (int)fileName.size(), fileName.data(), line, column, decl_count, (int)declaration->kind(), tk.offset());fflush(stdout);
There is a way to get the token usage location ?
@mingodad Thanks for sharing your changes, I will have a look at them this weekend.
Getting the ranges of the declarations using first/last SourceLocation should work, that is unless the declarations are generated from macro expansions, in such cases the preprocessor may assign token positions from the macro definitions, this is bug, I should probably fix this.
I wrote a simple Web app to test the tokens and AST locations. Unfortunately the current JavaScript API does not have support to resolve #include directives (I'm working on it), so you will need to run the preprocessor before using it.
cxx -E -P sqlite3.c -toolchain linux -o sqlite3.i
then paste the content of sqlite3.i to https://robertoraggi.github.io/cplusplus/?path=/story/cxxfrontend-syntaxtree--basic move the text cursor (or edit the C++ code), the AST browser will sync with the text editor.
With the latest changes I'm getting errors when trying to parse sqlite3.c:
/usr/bin/time mycxx -toolchain linux -fsyntax-only sqlite3.c > /dev/null
/usr/include/c++/11/type_traits:3391:23: expected a declaration
requires __is_enum(_Tp)
^
/usr/include/c++/11/type_traits:3391:27: expected a declaration
requires __is_enum(_Tp)
^
/usr/include/c++/11/type_traits:3392:5: expected a declaration
&& requires(_Tp __t) { __t = __t; } // fails if incomplete
^
/usr/include/c++/11/type_traits:3392:8: expected a declaration
&& requires(_Tp __t) { __t = __t; } // fails if incomplete
^
/usr/include/c++/11/type_traits:3392:16: expected a declaration
&& requires(_Tp __t) { __t = __t; } // fails if incomplete
^
/usr/include/c++/11/type_traits:3392:24: expected a declaration
&& requires(_Tp __t) { __t = __t; } // fails if incomplete
^
/usr/include/c++/11/type_traits:3392:26: expected a declaration
&& requires(_Tp __t) { __t = __t; } // fails if incomplete
^
/usr/include/c++/11/type_traits:3392:32: expected a declaration
&& requires(_Tp __t) { __t = __t; } // fails if incomplete
^
/usr/include/c++/11/type_traits:3392:37: expected a declaration
&& requires(_Tp __t) { __t = __t; } // fails if incomplete
^
/usr/include/c++/11/type_traits:3400:23: expected a declaration
requires __is_enum(_Tp)
^
/usr/include/c++/11/type_traits:3400:27: expected a declaration
requires __is_enum(_Tp)
^
/usr/include/c++/11/type_traits:3401:5: expected a declaration
&& requires(_Tp __t) { __t = __t; } // fails if incomplete
^
/usr/include/c++/11/type_traits:3401:8: expected a declaration
&& requires(_Tp __t) { __t = __t; } // fails if incomplete
^
/usr/include/c++/11/type_traits:3401:16: expected a declaration
&& requires(_Tp __t) { __t = __t; } // fails if incomplete
^
/usr/include/c++/11/type_traits:3401:24: expected a declaration
&& requires(_Tp __t) { __t = __t; } // fails if incomplete
^
/usr/include/c++/11/type_traits:3401:26: expected a declaration
&& requires(_Tp __t) { __t = __t; } // fails if incomplete
^
/usr/include/c++/11/type_traits:3401:32: expected a declaration
&& requires(_Tp __t) { __t = __t; } // fails if incomplete
^
/usr/include/c++/11/type_traits:3401:37: expected a declaration
&& requires(_Tp __t) { __t = __t; } // fails if incomplete
^
/usr/include/c++/11/type_traits:3599:1: expected a declaration
} // namespace std
^
/usr/include/c++/11/cmath:1938:1: expected a declaration
} // extern "C++"
^
Which version of sqlite are you using? I downloaded a random version, and it appears to parse correctly with gcc 11 and 14 headers, as long as I pass the -Dregister=“” to cxx.
I isolated the problem with this test: test-cmath.cpp
#include <cmath>
Testing with cxx -toolchain linux -fsyntax-only test-cmath.cpp gives the same output as reported here https://github.com/robertoraggi/cplusplus/issues/363#issuecomment-2470320209 .
I'm attaching (see bellow) the output of:
g++9 -E test-cmath.cpp > cmath-g++9.i
g++14 -E test-cmath.cpp > cmath-g++14.i
clang++19 -E test-cmath.cpp > cmath-clang++19.i
cxx -toolchain linux -E test-cmath.cpp > cmath-cxx.i
Also when trying to parse the preprocessed files cxx can't parse the one generated by itself:
cxx -toolchain linux -fsyntax-only cmath-cxx.i
cmath-cxx.i:3579:5: expected a name
[ [ deprecated ( "use is_standard_layout && is_trivial instead") ] ]
^
...
-------
cxx -toolchain linux -fsyntax-only cmath-g++9.i
#parse ok
-------
cxx -toolchain linux -fsyntax-only cmath-g++14.i
cmath-g++14.i:9121:39: expected ';'
{ using type = __type_pack_element<_Np, _Types...>; };
^
...
-------
cxx -toolchain linux -fsyntax-only cmath-clang++19.i
#parse ok
cxx -toolchain linux -fsyntax-only cmath-cxx.i
cmath-cxx.i:3579:5: expected a name
[ [ deprecated ( "use is_standard_layout && is_trivial instead") ] ]
This is the parser being too strict and not allowing spaces between brackets in C++ attributes. There are instances where the preprocessor introduces spaces, which can break the parsing of the preprocessor’s output. It should be fixed in https://github.com/robertoraggi/cplusplus/pull/416
You can’t parse the output preprocessed by gcc with cxx because gcc has a few built-in functions that I haven’t implemented yet (for example, __reference_constructs_from_temporary and __type_pack_element).
(https://github.com/robertoraggi/cplusplus/blob/main/.github/workflows/ci.yml#L54)
I have a test in the CI pipeline for this. I’m sure that cmath is included when parsing the repository.
On the other hand, parsing the output of cxx's preprocessor with gcc might work, but it all depends on which built-in (tested with __has_builtin) gcc is expecting. You may need to pass -w to gcc to avoid reporting warning from system headers.
echo '#include <iostream>' | ./build/src/frontend/cxx -toolchain linux -E - | g++ -fsyntax-only -xc++ -std=c++26 -w -
The parse error when using the __is_enum builtin traits in a require clause should be fixed in https://github.com/robertoraggi/cplusplus/pull/434 Thanks for the report.
Testing again with the latest code I've noticed that memory usage has increased more than 50%:
/usr/bin/time cppcxx -toolchain linux sqlite3.c -c -I.
1.98user 0.08system 0:02.09elapsed 98%CPU (0avgtext+0avgdata 262700maxresident)k
18520inputs+0outputs (0major+93756minor)pagefaults 0swaps
interesting, thanks for the report I will have a look. Maybe it is related to the change that replaces the arena with a std::monotonic_buffer_resource. I'm not seeing big changes with macOS but I do see some difference in the memory usage when I try with docker.
docker run --rm -it -u vscode -v $PWD:/code -w /code mcr.microsoft.com/devcontainers/cpp:ubuntu-24.04
cmake --preset default
cmake --build build
valgrind ./build/src/frontend/cxx tests/manual/source.cc
==1887== HEAP SUMMARY:
==1887== in use at exit: 0 bytes in 0 blocks
==1887== total heap usage: 7,030,751 allocs, 7,030,751 frees, 764,615,492 bytes allocated
valgrind --tool=callgrind ./build/src/frontend/cxx tests/manual/source.cc
==1898==
==1898== I refs: 6,171,184,464
git revert af85da3bca05ea99694a44b4226066840456e81c --no-commit
cmake --build build
valgrind ./build/src/frontend/cxx tests/manual/source.cc
==2613== in use at exit: 0 bytes in 0 blocks
==2613== total heap usage: 7,061,691 allocs, 7,061,691 frees, 728,041,300 bytes allocated
valgrind --tool=callgrind ./build/src/frontend/cxx tests/manual/source.cc
==2624==
==2624== I refs: 6,174,705,434
Also, make sure you start the cxx tool from a terminal (e.g. not from an IDE). I noticed that it's about 10% slower when I run cxx from the vscode terminal.
I'm experimenting with a permissive toolchain linuxC (see bellow changes) and when trying to parse sqlite3.c with -ast-print I'm getting an almost compilabe result except for wchar_t (see bellow) I'm looking for a way to exclude wchar_t as keyword from the lexer if using linuxC toolchain (probably other keywords would need be excluded if we accept -std=cxxYY too).
cxx -ast-print -toolchain linuxC sqlite3.c -c -I.
/usr/lib/gcc/x86_64-linux-gnu/11/include/stddef.h:321:24: error: expected a declaration
typedef __WCHAR_TYPE__ wchar_t;
^
/usr/lib/gcc/x86_64-linux-gnu/11/include/stddef.h:321:31: error: expected a declaration
typedef __WCHAR_TYPE__ wchar_t;
^
Diff to add linuxC toolchain:
------------------------- src/frontend/cxx/frontend.cc -------------------------
index 3565f261..195e0d0f 100644
@@ -25,6 +25,7 @@
#include <cxx/cli.h>
#include <cxx/control.h>
#include <cxx/gcc_linux_toolchain.h>
+#include <cxx/gcc_c_linux_toolchain.h>
#include <cxx/lexer.h>
#include <cxx/lsp/lsp_server.h>
#include <cxx/macos_toolchain.h>
@@ -203,6 +204,16 @@ auto runOnFile(const CLI& cli, const std::string& fileName) -> bool {
}
toolchain = std::move(wasmToolchain);
+ } else if (toolchainId == "linuxC") {
+ std::string host;
+#ifdef __aarch64__
+ host = "aarch64";
+#elif __x86_64__
+ host = "x86_64";
+#endif
+
+ std::string arch = cli.getSingle("-arch").value_or(host);
+ toolchain = std::make_unique<GCC_C_LinuxToolchain>(preprocessor, arch);
} else if (toolchainId == "linux") {
std::string host;
#ifdef __aarch64__
-------------------------- src/parser/CMakeLists.txt --------------------------
index 01b7e6d5..4bb922a3 100644
@@ -41,6 +41,7 @@ add_library(cxx-parser
cxx/diagnostics_client.cc
cxx/external_name_encoder.cc
cxx/gcc_linux_toolchain.cc
+ cxx/gcc_c_linux_toolchain.cc
cxx/lexer.cc
cxx/literals.cc
cxx/macos_toolchain.cc
--------------------- src/parser/cxx/gcc_linux_toolchain.h ---------------------
index 5e7ed75a..6b85ef86 100644
@@ -27,7 +27,7 @@
namespace cxx {
-class GCCLinuxToolchain final : public Toolchain {
+class GCCLinuxToolchain : public Toolchain {
public:
explicit GCCLinuxToolchain(Preprocessor* preprocessor,
std::string arch = "x86_64");
------------------------- src/parser/cxx/toolchain.cc -------------------------
index dfc94fda..1d7598c8 100644
@@ -41,6 +41,10 @@ void Toolchain::defineMacro(const std::string& name,
preprocessor_->defineMacro(name, definition);
}
+void Toolchain::undefMacro(const std::string& name) {
+ preprocessor_->undefMacro(name);
+}
+
void Toolchain::addSystemIncludePath(std::string path) {
preprocessor_->addSystemIncludePath(std::move(path));
}
-------------------------- src/parser/cxx/toolchain.h --------------------------
index 2c346d84..c5bef775 100644
@@ -53,6 +53,7 @@ class Toolchain {
}
void defineMacro(const std::string &name, const std::string &definition);
+ void undefMacro(const std::string &name);
void addSystemIncludePath(std::string path);
gcc_c_linux_toolchain.h:
// Copyright (c) 2025 Roberto Raggi <[email protected]>
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in
// all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
// SOFTWARE.
#pragma once
#include <cxx/gcc_linux_toolchain.h>
namespace cxx {
class GCC_C_LinuxToolchain final : public GCCLinuxToolchain {
public:
explicit GCC_C_LinuxToolchain(Preprocessor* preprocessor,
std::string arch = "x86_64");
void addSystemCppIncludePaths() override;
void addPredefinedMacros() override;
};
} // namespace cxx
gcc_linux_toolchain.cpp:
// Copyright (c) 2025 Roberto Raggi <[email protected]>
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in
// all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
// SOFTWARE.
#include <cxx/gcc_c_linux_toolchain.h>
#include <cxx/preprocessor.h>
#include <cxx/private/path.h>
namespace cxx {
GCC_C_LinuxToolchain::GCC_C_LinuxToolchain(Preprocessor* preprocessor,
std::string arch)
: GCCLinuxToolchain(preprocessor, arch) {
}
void GCC_C_LinuxToolchain::addSystemCppIncludePaths() {
}
void GCC_C_LinuxToolchain::addPredefinedMacros() {
GCCLinuxToolchain::addPredefinedMacros();
// remove C++ macros
undefMacro("__null");
defineMacro("__null", "(void*)0");
undefMacro("__cplusplus");
undefMacro("__EXCEPTIONS");
undefMacro("__GXX_ABI_VERSION");
undefMacro("__GXX_EXPERIMENTAL_CXX0X__");
undefMacro("__GXX_RTTI");
undefMacro("__GXX_WEAK__");
undefMacro("__cplusplus");
undefMacro("__cpp_aggregate_bases");
undefMacro("__cpp_aggregate_nsdmi");
undefMacro("__cpp_aggregate_paren_init");
undefMacro("__cpp_alias_templates");
undefMacro("__cpp_aligned_new");
undefMacro("__cpp_attributes");
undefMacro("__cpp_auto_cast");
undefMacro("__cpp_binary_literals");
undefMacro("__cpp_capture_star_this");
undefMacro("__cpp_char8_t");
undefMacro("__cpp_concepts");
undefMacro("__cpp_conditional_explicit");
undefMacro("__cpp_consteval");
undefMacro("__cpp_constexpr");
undefMacro("__cpp_constexpr_dynamic_alloc");
undefMacro("__cpp_constexpr_in_decltype");
undefMacro("__cpp_constinit");
undefMacro("__cpp_decltype");
undefMacro("__cpp_decltype_auto");
undefMacro("__cpp_deduction_guides");
undefMacro("__cpp_delegating_constructors");
undefMacro("__cpp_designated_initializers");
undefMacro("__cpp_digit_separators");
undefMacro("__cpp_enumerator_attributes");
undefMacro("__cpp_exceptions");
undefMacro("__cpp_explicit_this_parameter");
undefMacro("__cpp_fold_expressions");
undefMacro("__cpp_generic_lambdas");
undefMacro("__cpp_guaranteed_copy_elision");
undefMacro("__cpp_hex_float");
undefMacro("__cpp_if_consteval");
undefMacro("__cpp_if_constexpr");
undefMacro("__cpp_impl_coroutine");
undefMacro("__cpp_impl_destroying_delete");
undefMacro("__cpp_impl_three_way_comparison");
undefMacro("__cpp_implicit_move");
undefMacro("__cpp_inheriting_constructors");
undefMacro("__cpp_init_captures");
undefMacro("__cpp_initializer_lists");
undefMacro("__cpp_inline_variables");
undefMacro("__cpp_lambdas");
undefMacro("__cpp_multidimensional_subscript");
undefMacro("__cpp_named_character_escapes");
undefMacro("__cpp_namespace_attributes");
undefMacro("__cpp_nested_namespace_definitions");
undefMacro("__cpp_noexcept_function_type");
undefMacro("__cpp_nontype_template_args");
undefMacro("__cpp_nontype_template_parameter_auto");
undefMacro("__cpp_nontype_template_parameter_class");
undefMacro("__cpp_nsdmi");
undefMacro("__cpp_placeholder_variables");
undefMacro("__cpp_range_based_for");
undefMacro("__cpp_raw_strings");
undefMacro("__cpp_ref_qualifiers");
undefMacro("__cpp_return_type_deduction");
undefMacro("__cpp_rtti");
undefMacro("__cpp_runtime_arrays");
undefMacro("__cpp_rvalue_reference");
undefMacro("__cpp_rvalue_references");
undefMacro("__cpp_size_t_suffix");
undefMacro("__cpp_sized_deallocation");
undefMacro("__cpp_static_assert");
undefMacro("__cpp_static_call_operator");
undefMacro("__cpp_structured_bindings");
undefMacro("__cpp_template_auto");
undefMacro("__cpp_template_template_args");
undefMacro("__cpp_threadsafe_static_init");
undefMacro("__cpp_unicode_characters");
undefMacro("__cpp_unicode_literals");
undefMacro("__cpp_user_defined_literals");
undefMacro("__cpp_using_enum");
undefMacro("__cpp_variable_templates");
undefMacro("__cpp_variadic_templates");
undefMacro("__cpp_variadic_using");
}
} // namespace cxx
I've reworked my changes to linuxC toolchain and now the only missing piece to parse sqlite3.c cleanly is register keyword (see bellow the changes) but the output already pass this gcc -fsyntax-only sqlite3-ast-printC.c:
cxx -ast-print -toolchain linuxC /home/mingo/dev/dadbiz++/third-party/dad/fossil0/extsrc/sqlite3.c
/home/mingo/dev/dadbiz++/third-party/dad/fossil0/extsrc/sqlite3.c:35971:12: error: expected ';'
register unsigned char *a, *b;
^
typedef __builtin_va_list __gnuc_va_list;
typedef __gnuc_va_list va_list;
...
------------------------- src/frontend/cxx/frontend.cc -------------------------
index 3565f261..ad42af75 100644
@@ -25,6 +25,7 @@
#include <cxx/cli.h>
#include <cxx/control.h>
#include <cxx/gcc_linux_toolchain.h>
+#include <cxx/gcc_c_linux_toolchain.h>
#include <cxx/lexer.h>
#include <cxx/lsp/lsp_server.h>
#include <cxx/macos_toolchain.h>
@@ -116,7 +117,8 @@ auto readAll(const std::string& fileName) -> std::optional<std::string> {
return std::nullopt;
}
-void dumpTokens(const CLI& cli, TranslationUnit& unit, std::ostream& output) {
+void dumpTokens(const CLI& cli, TranslationUnit& unit, std::ostream& output,
+ bool onlyC=false) {
std::string flags;
for (SourceLocation loc(1);; loc = loc.next()) {
@@ -134,7 +136,7 @@ void dumpTokens(const CLI& cli, TranslationUnit& unit, std::ostream& output) {
auto kind = tk.kind();
if (kind == TokenKind::T_IDENTIFIER) {
- kind = Lexer::classifyKeyword(tk.spell());
+ kind = Lexer::classifyKeyword(tk.spell(), onlyC);
}
output << std::format("{} '{}'{}", Token::name(kind), tk.spell(), flags);
@@ -203,7 +205,7 @@ auto runOnFile(const CLI& cli, const std::string& fileName) -> bool {
}
toolchain = std::move(wasmToolchain);
- } else if (toolchainId == "linux") {
+ } else if (toolchainId == "linux" || toolchainId == "linuxC") {
std::string host;
#ifdef __aarch64__
host = "aarch64";
@@ -212,7 +214,10 @@ auto runOnFile(const CLI& cli, const std::string& fileName) -> bool {
#endif
std::string arch = cli.getSingle("-arch").value_or(host);
- toolchain = std::make_unique<GCCLinuxToolchain>(preprocessor, arch);
+ if (toolchainId == "linuxC")
+ toolchain = std::make_unique<GCC_C_LinuxToolchain>(preprocessor, arch);
+ else
+ toolchain = std::make_unique<GCCLinuxToolchain>(preprocessor, arch);
} else if (toolchainId == "windows") {
auto windowsToolchain = std::make_unique<WindowsToolchain>(preprocessor);
-------------------- src/parser/cxx/gcc_linux_toolchain.cc --------------------
index c5140270..854ef9f4 100644
@@ -27,8 +27,8 @@
namespace cxx {
GCCLinuxToolchain::GCCLinuxToolchain(Preprocessor* preprocessor,
- std::string arch)
- : Toolchain(preprocessor), arch_(std::move(arch)) {
+ std::string arch, bool onlyC)
+ : Toolchain(preprocessor), arch_(std::move(arch)), onlyC_(onlyC) {
for (int version : {15, 14, 13, 12, 11, 10, 9}) {
const auto path = fs::path(
std::format("/usr/lib/gcc/{}-linux-gnu/{}/include", arch_, version));
@@ -54,6 +54,7 @@ void GCCLinuxToolchain::addSystemIncludePaths() {
}
void GCCLinuxToolchain::addSystemCppIncludePaths() {
+ if(onlyC_) return;
auto addSystemIncludePathForGCCVersion = [this](int version) {
addSystemIncludePath(std::format("/usr/include/c++/{}/backward", version));
@@ -68,7 +69,8 @@ void GCCLinuxToolchain::addSystemCppIncludePaths() {
void GCCLinuxToolchain::addPredefinedMacros() {
defineMacro("__extension__", "");
- defineMacro("__null", "nullptr");
+ if(onlyC_) defineMacro("__null", "(void*)0");
+ else defineMacro("__null", "nullptr");
defineMacro("__restrict__", "");
defineMacro("__restrict", "");
defineMacro("__signed__", "signed");
@@ -77,7 +79,7 @@ void GCCLinuxToolchain::addPredefinedMacros() {
defineMacro("_Pragma(x)", "");
// std=c++26
- defineMacro("__cplusplus", "202400L");
+ if(!onlyC_) defineMacro("__cplusplus", "202400L");
addCommonMacros();
@@ -162,7 +164,7 @@ void GCCLinuxToolchain::addCommonMacros() {
defineMacro("__DEC_EVAL_METHOD__", "2");
defineMacro("__DEPRECATED", "1");
defineMacro("__ELF__", "1");
- defineMacro("__EXCEPTIONS", "1");
+ if(!onlyC_) defineMacro("__EXCEPTIONS", "1");
defineMacro("__FINITE_MATH_ONLY__", "0");
defineMacro("__FLOAT_WORD_ORDER__", "__ORDER_LITTLE_ENDIAN__");
defineMacro("__FLT128_DECIMAL_DIG__", "36");
@@ -301,10 +303,12 @@ void GCCLinuxToolchain::addCommonMacros() {
defineMacro("__GNUC_WIDE_EXECUTION_CHARSET_NAME", "\"UTF-32LE\"");
defineMacro("__GNUC__", "14");
defineMacro("__GNUG__", "14");
- defineMacro("__GXX_ABI_VERSION", "1019");
- defineMacro("__GXX_EXPERIMENTAL_CXX0X__", "1");
- defineMacro("__GXX_RTTI", "1");
- defineMacro("__GXX_WEAK__", "1");
+ if(!onlyC_) {
+ defineMacro("__GXX_ABI_VERSION", "1019");
+ defineMacro("__GXX_EXPERIMENTAL_CXX0X__", "1");
+ defineMacro("__GXX_RTTI", "1");
+ defineMacro("__GXX_WEAK__", "1");
+ }
defineMacro("__HAVE_SPECULATION_SAFE_VALUE", "1");
defineMacro("__INT16_C(c)", "c");
defineMacro("__INT16_MAX__", "0x7fff");
@@ -455,81 +459,83 @@ void GCCLinuxToolchain::addCommonMacros() {
defineMacro("__WINT_MIN__", "0U");
defineMacro("__WINT_TYPE__", "unsigned int");
defineMacro("__WINT_WIDTH__", "32");
- defineMacro("__cplusplus", "202400L");
- defineMacro("__cpp_aggregate_bases", "201603L");
- defineMacro("__cpp_aggregate_nsdmi", "201304L");
- defineMacro("__cpp_aggregate_paren_init", "201902L");
- defineMacro("__cpp_alias_templates", "200704L");
- defineMacro("__cpp_aligned_new", "201606L");
- defineMacro("__cpp_attributes", "200809L");
- defineMacro("__cpp_auto_cast", "202110L");
- defineMacro("__cpp_binary_literals", "201304L");
- defineMacro("__cpp_capture_star_this", "201603L");
- defineMacro("__cpp_char8_t", "202207L");
- defineMacro("__cpp_concepts", "202002L");
- defineMacro("__cpp_conditional_explicit", "201806L");
- defineMacro("__cpp_consteval", "202211L");
- defineMacro("__cpp_constexpr", "202306L");
- defineMacro("__cpp_constexpr_dynamic_alloc", "201907L");
- defineMacro("__cpp_constexpr_in_decltype", "201711L");
- defineMacro("__cpp_constinit", "201907L");
- defineMacro("__cpp_decltype", "200707L");
- defineMacro("__cpp_decltype_auto", "201304L");
- defineMacro("__cpp_deduction_guides", "201907L");
- defineMacro("__cpp_delegating_constructors", "200604L");
- defineMacro("__cpp_designated_initializers", "201707L");
- defineMacro("__cpp_digit_separators", "201309L");
- defineMacro("__cpp_enumerator_attributes", "201411L");
- defineMacro("__cpp_exceptions", "199711L");
- defineMacro("__cpp_explicit_this_parameter", "202110L");
- defineMacro("__cpp_fold_expressions", "201603L");
- defineMacro("__cpp_generic_lambdas", "201707L");
- defineMacro("__cpp_guaranteed_copy_elision", "201606L");
- defineMacro("__cpp_hex_float", "201603L");
- defineMacro("__cpp_if_consteval", "202106L");
- defineMacro("__cpp_if_constexpr", "201606L");
- defineMacro("__cpp_impl_coroutine", "201902L");
- defineMacro("__cpp_impl_destroying_delete", "201806L");
- defineMacro("__cpp_impl_three_way_comparison", "201907L");
- defineMacro("__cpp_implicit_move", "202207L");
- defineMacro("__cpp_inheriting_constructors", "201511L");
- defineMacro("__cpp_init_captures", "201803L");
- defineMacro("__cpp_initializer_lists", "200806L");
- defineMacro("__cpp_inline_variables", "201606L");
- defineMacro("__cpp_lambdas", "200907L");
- defineMacro("__cpp_multidimensional_subscript", "202211L");
- defineMacro("__cpp_named_character_escapes", "202207L");
- defineMacro("__cpp_namespace_attributes", "201411L");
- defineMacro("__cpp_nested_namespace_definitions", "201411L");
- defineMacro("__cpp_noexcept_function_type", "201510L");
- defineMacro("__cpp_nontype_template_args", "201911L");
- defineMacro("__cpp_nontype_template_parameter_auto", "201606L");
- defineMacro("__cpp_nontype_template_parameter_class", "201806L");
- defineMacro("__cpp_nsdmi", "200809L");
- defineMacro("__cpp_placeholder_variables", "202306L");
- defineMacro("__cpp_range_based_for", "201603L");
- defineMacro("__cpp_raw_strings", "200710L");
- defineMacro("__cpp_ref_qualifiers", "200710L");
- defineMacro("__cpp_return_type_deduction", "201304L");
- defineMacro("__cpp_rtti", "199711L");
- defineMacro("__cpp_runtime_arrays", "198712L");
- defineMacro("__cpp_rvalue_reference", "200610L");
- defineMacro("__cpp_rvalue_references", "200610L");
- defineMacro("__cpp_size_t_suffix", "202011L");
- defineMacro("__cpp_sized_deallocation", "201309L");
- defineMacro("__cpp_static_assert", "202306L");
- defineMacro("__cpp_static_call_operator", "202207L");
- defineMacro("__cpp_structured_bindings", "201606L");
- defineMacro("__cpp_template_auto", "201606L");
- defineMacro("__cpp_template_template_args", "201611L");
- defineMacro("__cpp_threadsafe_static_init", "200806L");
- defineMacro("__cpp_unicode_characters", "201411L");
- defineMacro("__cpp_unicode_literals", "200710L");
- defineMacro("__cpp_user_defined_literals", "200809L");
- defineMacro("__cpp_using_enum", "201907L");
- defineMacro("__cpp_variable_templates", "201304L");
- defineMacro("__cpp_variadic_templates", "200704L");
- defineMacro("__cpp_variadic_using", "201611L");
+ if(!onlyC_) {
+ defineMacro("__cplusplus", "202400L");
+ defineMacro("__cpp_aggregate_bases", "201603L");
+ defineMacro("__cpp_aggregate_nsdmi", "201304L");
+ defineMacro("__cpp_aggregate_paren_init", "201902L");
+ defineMacro("__cpp_alias_templates", "200704L");
+ defineMacro("__cpp_aligned_new", "201606L");
+ defineMacro("__cpp_attributes", "200809L");
+ defineMacro("__cpp_auto_cast", "202110L");
+ defineMacro("__cpp_binary_literals", "201304L");
+ defineMacro("__cpp_capture_star_this", "201603L");
+ defineMacro("__cpp_char8_t", "202207L");
+ defineMacro("__cpp_concepts", "202002L");
+ defineMacro("__cpp_conditional_explicit", "201806L");
+ defineMacro("__cpp_consteval", "202211L");
+ defineMacro("__cpp_constexpr", "202306L");
+ defineMacro("__cpp_constexpr_dynamic_alloc", "201907L");
+ defineMacro("__cpp_constexpr_in_decltype", "201711L");
+ defineMacro("__cpp_constinit", "201907L");
+ defineMacro("__cpp_decltype", "200707L");
+ defineMacro("__cpp_decltype_auto", "201304L");
+ defineMacro("__cpp_deduction_guides", "201907L");
+ defineMacro("__cpp_delegating_constructors", "200604L");
+ defineMacro("__cpp_designated_initializers", "201707L");
+ defineMacro("__cpp_digit_separators", "201309L");
+ defineMacro("__cpp_enumerator_attributes", "201411L");
+ defineMacro("__cpp_exceptions", "199711L");
+ defineMacro("__cpp_explicit_this_parameter", "202110L");
+ defineMacro("__cpp_fold_expressions", "201603L");
+ defineMacro("__cpp_generic_lambdas", "201707L");
+ defineMacro("__cpp_guaranteed_copy_elision", "201606L");
+ defineMacro("__cpp_hex_float", "201603L");
+ defineMacro("__cpp_if_consteval", "202106L");
+ defineMacro("__cpp_if_constexpr", "201606L");
+ defineMacro("__cpp_impl_coroutine", "201902L");
+ defineMacro("__cpp_impl_destroying_delete", "201806L");
+ defineMacro("__cpp_impl_three_way_comparison", "201907L");
+ defineMacro("__cpp_implicit_move", "202207L");
+ defineMacro("__cpp_inheriting_constructors", "201511L");
+ defineMacro("__cpp_init_captures", "201803L");
+ defineMacro("__cpp_initializer_lists", "200806L");
+ defineMacro("__cpp_inline_variables", "201606L");
+ defineMacro("__cpp_lambdas", "200907L");
+ defineMacro("__cpp_multidimensional_subscript", "202211L");
+ defineMacro("__cpp_named_character_escapes", "202207L");
+ defineMacro("__cpp_namespace_attributes", "201411L");
+ defineMacro("__cpp_nested_namespace_definitions", "201411L");
+ defineMacro("__cpp_noexcept_function_type", "201510L");
+ defineMacro("__cpp_nontype_template_args", "201911L");
+ defineMacro("__cpp_nontype_template_parameter_auto", "201606L");
+ defineMacro("__cpp_nontype_template_parameter_class", "201806L");
+ defineMacro("__cpp_nsdmi", "200809L");
+ defineMacro("__cpp_placeholder_variables", "202306L");
+ defineMacro("__cpp_range_based_for", "201603L");
+ defineMacro("__cpp_raw_strings", "200710L");
+ defineMacro("__cpp_ref_qualifiers", "200710L");
+ defineMacro("__cpp_return_type_deduction", "201304L");
+ defineMacro("__cpp_rtti", "199711L");
+ defineMacro("__cpp_runtime_arrays", "198712L");
+ defineMacro("__cpp_rvalue_reference", "200610L");
+ defineMacro("__cpp_rvalue_references", "200610L");
+ defineMacro("__cpp_size_t_suffix", "202011L");
+ defineMacro("__cpp_sized_deallocation", "201309L");
+ defineMacro("__cpp_static_assert", "202306L");
+ defineMacro("__cpp_static_call_operator", "202207L");
+ defineMacro("__cpp_structured_bindings", "201606L");
+ defineMacro("__cpp_template_auto", "201606L");
+ defineMacro("__cpp_template_template_args", "201611L");
+ defineMacro("__cpp_threadsafe_static_init", "200806L");
+ defineMacro("__cpp_unicode_characters", "201411L");
+ defineMacro("__cpp_unicode_literals", "200710L");
+ defineMacro("__cpp_user_defined_literals", "200809L");
+ defineMacro("__cpp_using_enum", "201907L");
+ defineMacro("__cpp_variable_templates", "201304L");
+ defineMacro("__cpp_variadic_templates", "200704L");
+ defineMacro("__cpp_variadic_using", "201611L");
+ }
defineMacro("__gnu_linux__", "1");
defineMacro("__linux", "1");
defineMacro("__linux__", "1");
--------------------- src/parser/cxx/gcc_linux_toolchain.h ---------------------
index 5e7ed75a..e0495b0e 100644
@@ -27,10 +27,10 @@
namespace cxx {
-class GCCLinuxToolchain final : public Toolchain {
+class GCCLinuxToolchain : public Toolchain {
public:
explicit GCCLinuxToolchain(Preprocessor* preprocessor,
- std::string arch = "x86_64");
+ std::string arch = "x86_64", bool onlyC = false);
[[nodiscard]] auto version() const -> std::optional<int> { return version_; }
@@ -45,6 +45,7 @@ class GCCLinuxToolchain final : public Toolchain {
private:
std::optional<int> version_;
std::string arch_;
+ bool onlyC_;
};
} // namespace cxx
--------------------------- src/parser/cxx/lexer.cc ---------------------------
index 467f9457..4431f68f 100644
@@ -107,17 +107,18 @@ inline auto skipBOM(It& it, It end) -> bool {
}
} // namespace
-Lexer::Lexer(std::string_view source)
- : source_(source), pos_(cbegin(source_)), end_(cend(source_)) {
+Lexer::Lexer(std::string_view source, bool onlyC)
+ : source_(source), pos_(cbegin(source_)),
+ end_(cend(source_)), onlyC_(onlyC) {
hasBOM_ = skipBOM(pos_, end_);
currentChar_ = pos_ < end_ ? peekNext(pos_, end_) : 0;
}
-Lexer::Lexer(std::string buffer)
+Lexer::Lexer(std::string buffer, bool onlyC)
: buffer_(std::move(buffer)),
source_(buffer_),
pos_(cbegin(source_)),
- end_(cend(source_)) {
+ end_(cend(source_)), onlyC_(onlyC) {
hasBOM_ = skipBOM(pos_, end_);
currentChar_ = pos_ < end_ ? peekNext(pos_, end_) : 0;
}
@@ -206,13 +207,7 @@ auto Lexer::readToken() -> TokenKind {
if (preprocessing_) return TokenKind::T_IDENTIFIER;
- if (auto keyword =
- classify(text_.c_str(), static_cast<int>(text_.length()));
- keyword != TokenKind::T_IDENTIFIER) {
- return keyword;
- }
-
- return TokenKind::T_IDENTIFIER;
+ return Lexer::classifyKeyword(text_, onlyC_);
}
}
@@ -259,7 +254,7 @@ auto Lexer::readToken() -> TokenKind {
}
bool ud = false;
- if (std::isalpha(LA()) || LA() == '_') {
+ if (!onlyC_ && std::isalpha(LA()) || LA() == '_') {
ud = true;
do {
consume();
@@ -569,8 +564,16 @@ auto Lexer::skipSpaces() -> bool {
return pos_ != end_;
}
-auto Lexer::classifyKeyword(const std::string_view& text) -> TokenKind {
- return classify(text.data(), static_cast<int>(text.size()));
+auto Lexer::classifyKeyword(const std::string_view& text, bool onlyC) -> TokenKind {
+ cxx::TokenKind rc = classify(text.data(), static_cast<int>(text.size()));
+ if(onlyC) {
+ switch(rc) {
+ case cxx::TokenKind::T_WCHAR_T:
+ case cxx::TokenKind::T_NEW: return cxx::TokenKind::T_IDENTIFIER;
+ default: return rc;
+ }
+ }
+ return rc;
}
void Lexer::clearBuffer() { buffer_.clear(); }
---------------------------- src/parser/cxx/lexer.h ----------------------------
index d8989e92..a76fee33 100644
@@ -29,8 +29,8 @@ namespace cxx {
class Lexer {
public:
- explicit Lexer(std::string_view source);
- explicit Lexer(std::string buffer);
+ explicit Lexer(std::string_view source, bool onlyC=false);
+ explicit Lexer(std::string buffer, bool onlyC=false);
[[nodiscard]] auto preprocessing() const -> bool { return preprocessing_; }
void setPreprocessing(bool preprocessing) { preprocessing_ = preprocessing; }
@@ -77,7 +77,7 @@ class Lexer {
[[nodiscard]] auto text() -> std::string& { return text_; }
[[nodiscard]] auto text() const -> const std::string& { return text_; }
- static auto classifyKeyword(const std::string_view& text) -> TokenKind;
+ static auto classifyKeyword(const std::string_view& text, bool onlyC=false) -> TokenKind;
struct State {
std::string_view::const_iterator pos_;
@@ -128,6 +128,7 @@ class Lexer {
bool preprocessing_ = false;
bool hasBOM_ = false;
+ bool onlyC_ = false;
};
} // namespace cxx
------------------------ src/parser/cxx/preprocessor.cc ------------------------
index 99bcbe6f..991dcbbe 100644
@@ -733,6 +733,7 @@ struct Preprocessor::Private {
int counter_ = 0;
int includeDepth_ = 0;
bool omitLineMarkers_ = false;
+ bool onlyC_ = false;
Arena pool_;
Private();
@@ -1600,6 +1601,7 @@ void Preprocessor::Private::initialize() {
adddBuiltinFunctionMacro("__has_include", hasInclude);
adddBuiltinFunctionMacro("__has_include_next", hasInclude);
+ onlyC_ = !preprocessor_->definedMacro("__cplusplus");
}
void Preprocessor::Private::finalizeToken(std::vector<Token> &tokens,
@@ -1622,8 +1624,8 @@ void Preprocessor::Private::finalizeToken(std::vector<Token> &tokens,
switch (tk->kind) {
case TokenKind::T_IDENTIFIER: {
- kind = Lexer::classifyKeyword(tk->text);
-
+ kind = Lexer::classifyKeyword(tk->text, onlyC_);
+
if (kind == TokenKind::T_IDENTIFIER) {
value.idValue = control_->getIdentifier(tk->text);
}
@@ -1711,7 +1713,7 @@ void Preprocessor::Private::finalizeToken(std::vector<Token> &tokens,
auto Preprocessor::Private::tokenize(const std::string_view &source,
int sourceFile, bool bol) -> TokList * {
- cxx::Lexer lex(source);
+ cxx::Lexer lex(source, onlyC_);
lex.setKeepComments(true);
lex.setPreprocessing(true);
TokList *ts = nullptr;
@@ -3140,6 +3142,11 @@ void Preprocessor::defineMacro(const std::string &name,
d->defineMacro(tokens);
}
+bool Preprocessor::definedMacro(const std::string &name) {
+ auto it = d->macros_.find(name);
+ return it != d->macros_.end();
+}
+
void Preprocessor::undefMacro(const std::string &name) {
auto it = d->macros_.find(name);
if (it != d->macros_.end()) d->macros_.erase(it);
------------------------ src/parser/cxx/preprocessor.h ------------------------
index 05dc174f..08b604a4 100644
@@ -99,6 +99,8 @@ class Preprocessor {
void defineMacro(const std::string &name, const std::string &body);
+ bool definedMacro(const std::string &name);
+
void undefMacro(const std::string &name);
void printMacros(std::ostream &out) const;
------------------------- src/parser/cxx/toolchain.cc -------------------------
index dfc94fda..1d7598c8 100644
@@ -41,6 +41,10 @@ void Toolchain::defineMacro(const std::string& name,
preprocessor_->defineMacro(name, definition);
}
+void Toolchain::undefMacro(const std::string& name) {
+ preprocessor_->undefMacro(name);
+}
+
void Toolchain::addSystemIncludePath(std::string path) {
preprocessor_->addSystemIncludePath(std::move(path));
}
-------------------------- src/parser/cxx/toolchain.h --------------------------
index 2c346d84..c5bef775 100644
@@ -53,6 +53,7 @@ class Toolchain {
}
void defineMacro(const std::string &name, const std::string &definition);
+ void undefMacro(const std::string &name);
void addSystemIncludePath(std::string path);
gcc_c_linux_toolchain.h:
// Copyright (c) 2025 Roberto Raggi <[email protected]>
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in
// all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
// SOFTWARE.
#pragma once
#include <cxx/gcc_linux_toolchain.h>
namespace cxx {
class GCC_C_LinuxToolchain final : public GCCLinuxToolchain {
public:
explicit GCC_C_LinuxToolchain(Preprocessor* preprocessor,
std::string arch = "x86_64"):
GCCLinuxToolchain(preprocessor, arch, true) {};
};
} // namespace cxx
Now when I parse shell.c from sqlite3 I'm getting no errors/warnings:
cxx -ast-print -toolchain linuxC fossil0/extsrc/shell.c
typedef unsigned int u32;
typedef unsigned short int u16;
typedef long unsigned int size_t;
...
But then when trying gcc -fsyntax-only sqlite3-shell-ast-printC.c I'm getting several errors/warnings:
gcc -fsyntax-only sqlite3-shell-ast-printC.c
sqlite3-shell-ast-printC.c: In function ‘quoteChar’:
sqlite3-shell-ast-printC.c:6581:10: warning: implicit declaration of function ‘sqlite3_keyword_check’ [-Wimplicit-function-declaration]
6581 | return sqlite3_keyword_check(zName, i) ? '"': 0;
| ^~~~~~~~~~~~~~~~~~~~~
sqlite3-shell-ast-printC.c: In function ‘sqlite3_shathree_init’:
sqlite3-shell-ast-printC.c:7345:51: error: ‘SQLITE_INNOCUOUS’ undeclared (first use in this function)
7345 | rc = sqlite3_create_function(db, "sha3", 1, 1 | SQLITE_INNOCUOUS | 0x800, 0, sha3Func, 0, 0);
| ^~~~~~~~~~~~~~~~
But if I add an include command line -Ifossil0/extsrc then cxx still parses without errors/warnings and gcc -fsyntax-only sqlite3-shell-ast-printC.c pass.
See the diff of the -ast-print without/with -I:
--- <unnamed>
+++ <unnamed>
@@ -4159,6 +4159,7 @@
};
typedef struct sqlite3_mutex sqlite3_mutex;
typedef struct sqlite3_api_routines sqlite3_api_routines;
+typedef const char * sqlite3_filename;
typedef struct sqlite3_vfs sqlite3_vfs;
typedef void(* sqlite3_syscall_ptr)(void);
struct sqlite3_vfs {
@@ -4168,7 +4169,7 @@
sqlite3_vfs * pNext;
const char * zName;
void * pAppData;
- int(* xOpen)(sqlite3_vfs *, const char * zName, sqlite3_file *, int flags, int * pOutFlags);
+ int(* xOpen)(sqlite3_vfs *, sqlite3_filename zName, sqlite3_file *, int flags, int * pOutFlags);
int(* xDelete)(sqlite3_vfs *, const char * zName, int syncDir);
int(* xAccess)(sqlite3_vfs *, const char * zName, int flags, int * pResOut);
int(* xFullPathname)(sqlite3_vfs *, const char * zName, int nOut, char * zOut);
@@ -4206,12 +4207,16 @@
sqlite3_int64 sqlite3_last_insert_rowid(sqlite3 *);
void sqlite3_set_last_insert_rowid(sqlite3 *, sqlite3_int64);
int sqlite3_changes(sqlite3 *);
+sqlite3_int64 sqlite3_changes64(sqlite3 *);
int sqlite3_total_changes(sqlite3 *);
+sqlite3_int64 sqlite3_total_changes64(sqlite3 *);
void sqlite3_interrupt(sqlite3 *);
+int sqlite3_is_interrupted(sqlite3 *);
int sqlite3_complete(const char * sql);
int sqlite3_complete16(const void * sql);
int sqlite3_busy_handler(sqlite3 *, int(*)(void *, int), void *);
int sqlite3_busy_timeout(sqlite3 *, int ms);
+int sqlite3_setlk_timeout(sqlite3 *, int ms, int flags);
int sqlite3_get_table(sqlite3 * db, const char * zSql, char * * * pazResult, int * pnRow, int * pnColumn, char * * pzErrmsg);
void sqlite3_free_table(char * * result);
char * sqlite3_mprintf(const char *, ...);
@@ -4235,14 +4240,22 @@
int sqlite3_open(const char * filename, sqlite3 * * ppDb);
int sqlite3_open16(const void * filename, sqlite3 * * ppDb);
int sqlite3_open_v2(const char * filename, sqlite3 * * ppDb, int flags, const char * zVfs);
-const char * sqlite3_uri_parameter(const char * zFilename, const char * zParam);
-int sqlite3_uri_boolean(const char * zFile, const char * zParam, int bDefault);
-sqlite3_int64 sqlite3_uri_int64(const char *, const char *, sqlite3_int64);
+const char * sqlite3_uri_parameter(sqlite3_filename z, const char * zParam);
+int sqlite3_uri_boolean(sqlite3_filename z, const char * zParam, int bDefault);
+sqlite3_int64 sqlite3_uri_int64(sqlite3_filename, const char *, sqlite3_int64);
+const char * sqlite3_uri_key(sqlite3_filename z, int N);
+const char * sqlite3_filename_database(sqlite3_filename);
+const char * sqlite3_filename_journal(sqlite3_filename);
+const char * sqlite3_filename_wal(sqlite3_filename);
+sqlite3_file * sqlite3_database_file_object(const char *);
+sqlite3_filename sqlite3_create_filename(const char * zDatabase, const char * zJournal, const char * zWal, int nParam, const char * * azParam);
+void sqlite3_free_filename(sqlite3_filename);
int sqlite3_errcode(sqlite3 * db);
int sqlite3_extended_errcode(sqlite3 * db);
const char * sqlite3_errmsg(sqlite3 *);
const void * sqlite3_errmsg16(sqlite3 *);
const char * sqlite3_errstr(int);
+int sqlite3_error_offset(sqlite3 * db);
typedef struct sqlite3_stmt sqlite3_stmt;
int sqlite3_limit(sqlite3 *, int id, int newVal);
int sqlite3_prepare(sqlite3 * db, const char * zSql, int nByte, sqlite3_stmt * * ppStmt, const char * * pzTail);
@@ -4254,6 +4267,8 @@
const char * sqlite3_sql(sqlite3_stmt * pStmt);
char * sqlite3_expanded_sql(sqlite3_stmt * pStmt);
int sqlite3_stmt_readonly(sqlite3_stmt * pStmt);
+int sqlite3_stmt_isexplain(sqlite3_stmt * pStmt);
+int sqlite3_stmt_explain(sqlite3_stmt * pStmt, int eMode);
int sqlite3_stmt_busy(sqlite3_stmt *);
typedef struct sqlite3_value sqlite3_value;
typedef struct sqlite3_context sqlite3_context;
@@ -4302,6 +4317,7 @@
int sqlite3_create_function(sqlite3 * db, const char * zFunctionName, int nArg, int eTextRep, void * pApp, void(* xFunc)(sqlite3_context *, int, sqlite3_value * *), void(* xStep)(sqlite3_context *, int, sqlite3_value * *), void(* xFinal)(sqlite3_context *));
int sqlite3_create_function16(sqlite3 * db, const void * zFunctionName, int nArg, int eTextRep, void * pApp, void(* xFunc)(sqlite3_context *, int, sqlite3_value * *), void(* xStep)(sqlite3_context *, int, sqlite3_value * *), void(* xFinal)(sqlite3_context *));
int sqlite3_create_function_v2(sqlite3 * db, const char * zFunctionName, int nArg, int eTextRep, void * pApp, void(* xFunc)(sqlite3_context *, int, sqlite3_value * *), void(* xStep)(sqlite3_context *, int, sqlite3_value * *), void(* xFinal)(sqlite3_context *), void(* xDestroy)(void *));
+int sqlite3_create_window_function(sqlite3 * db, const char * zFunctionName, int nArg, int eTextRep, void * pApp, void(* xStep)(sqlite3_context *, int, sqlite3_value * *), void(* xFinal)(sqlite3_context *), void(* xValue)(sqlite3_context *), void(* xInverse)(sqlite3_context *, int, sqlite3_value * *), void(* xDestroy)(void *));
int sqlite3_aggregate_count(sqlite3_context *);
int sqlite3_expired(sqlite3_stmt *);
int sqlite3_transfer_bindings(sqlite3_stmt *, sqlite3_stmt *);
@@ -4322,6 +4338,8 @@
int sqlite3_value_type(sqlite3_value *);
int sqlite3_value_numeric_type(sqlite3_value *);
int sqlite3_value_nochange(sqlite3_value *);
+int sqlite3_value_frombind(sqlite3_value *);
+int sqlite3_value_encoding(sqlite3_value *);
unsigned int sqlite3_value_subtype(sqlite3_value *);
sqlite3_value * sqlite3_value_dup(const sqlite3_value *);
void sqlite3_value_free(sqlite3_value *);
@@ -4330,6 +4348,8 @@
sqlite3 * sqlite3_context_db_handle(sqlite3_context *);
void * sqlite3_get_auxdata(sqlite3_context *, int N);
void sqlite3_set_auxdata(sqlite3_context *, int N, void *, void(*)(void *));
+void * sqlite3_get_clientdata(sqlite3 *, const char *);
+int sqlite3_set_clientdata(sqlite3 *, const char *, void *, void(*)(void *));
typedef void(* sqlite3_destructor_type)(void *);
void sqlite3_result_blob(sqlite3_context *, const void *, int, void(*)(void *));
void sqlite3_result_blob64(sqlite3_context *, const void *, sqlite3_uint64, void(*)(void *));
@@ -4360,18 +4380,25 @@
int sqlite3_sleep(int);
extern char * sqlite3_temp_directory;
extern char * sqlite3_data_directory;
+int sqlite3_win32_set_directory(unsigned long type, void * zValue);
+int sqlite3_win32_set_directory8(unsigned long type, const char * zValue);
+int sqlite3_win32_set_directory16(unsigned long type, const void * zValue);
int sqlite3_get_autocommit(sqlite3 *);
sqlite3 * sqlite3_db_handle(sqlite3_stmt *);
-const char * sqlite3_db_filename(sqlite3 * db, const char * zDbName);
+const char * sqlite3_db_name(sqlite3 * db, int N);
+sqlite3_filename sqlite3_db_filename(sqlite3 * db, const char * zDbName);
int sqlite3_db_readonly(sqlite3 * db, const char * zDbName);
+int sqlite3_txn_state(sqlite3 *, const char * zSchema);
sqlite3_stmt * sqlite3_next_stmt(sqlite3 * pDb, sqlite3_stmt * pStmt);
void * sqlite3_commit_hook(sqlite3 *, int(*)(void *), void *);
void * sqlite3_rollback_hook(sqlite3 *, void(*)(void *), void *);
+int sqlite3_autovacuum_pages(sqlite3 * db, unsigned int(*)(void *, const char *, unsigned int, unsigned int, unsigned int), void *, void(*)(void *));
void * sqlite3_update_hook(sqlite3 *, void(*)(void *, int, char const *, char const *, sqlite3_int64), void *);
int sqlite3_enable_shared_cache(int);
int sqlite3_release_memory(int);
int sqlite3_db_release_memory(sqlite3 *);
sqlite3_int64 sqlite3_soft_heap_limit64(sqlite3_int64 N);
+sqlite3_int64 sqlite3_hard_heap_limit64(sqlite3_int64 N);
void sqlite3_soft_heap_limit(int N);
int sqlite3_table_column_metadata(sqlite3 * db, const char * zDbName, const char * zTableName, const char * zColumnName, char const * * pzDataType, char const * * pzCollSeq, int * pNotNull, int * pPrimaryKey, int * pAutoinc);
int sqlite3_load_extension(sqlite3 * db, const char * zFile, const char * zProc, char * * pzErrMsg);
@@ -4407,6 +4434,8 @@
int(* xSavepoint)(sqlite3_vtab * pVTab, int);
int(* xRelease)(sqlite3_vtab * pVTab, int);
int(* xRollbackTo)(sqlite3_vtab * pVTab, int);
+ int(* xShadowName)(const char *);
+ int(* xIntegrity)(sqlite3_vtab * pVTab, const char * zSchema, const char * zTabName, int mFlags, char * * pzErr);
};
struct sqlite3_index_info {
int nConstraint;
@@ -4439,6 +4468,7 @@
};
int sqlite3_create_module(sqlite3 * db, const char * zName, const sqlite3_module * p, void * pClientData);
int sqlite3_create_module_v2(sqlite3 * db, const char * zName, const sqlite3_module * p, void * pClientData, void(* xDestroy)(void *));
+int sqlite3_drop_modules(sqlite3 * db, const char * * azKeep);
struct sqlite3_vtab {
const sqlite3_module * pModule;
int nRef;
@@ -4481,6 +4511,9 @@
sqlite3_mutex * sqlite3_db_mutex(sqlite3 *);
int sqlite3_file_control(sqlite3 *, const char * zDbName, int op, void *);
int sqlite3_test_control(int op, ...);
+int sqlite3_keyword_count(void);
+int sqlite3_keyword_name(int, const char * *, int *);
+int sqlite3_keyword_check(const char *, int);
typedef struct sqlite3_str sqlite3_str;
sqlite3_str * sqlite3_str_new(sqlite3 *);
char * sqlite3_str_finish(sqlite3_str *);
@@ -4553,7 +4586,13 @@
int sqlite3_vtab_on_conflict(sqlite3 *);
int sqlite3_vtab_nochange(sqlite3_context *);
const char * sqlite3_vtab_collation(sqlite3_index_info *, int);
+int sqlite3_vtab_distinct(sqlite3_index_info *);
+int sqlite3_vtab_in(sqlite3_index_info *, int iCons, int bHandle);
+int sqlite3_vtab_in_first(sqlite3_value * pVal, sqlite3_value * * ppOut);
+int sqlite3_vtab_in_next(sqlite3_value * pVal, sqlite3_value * * ppOut);
+int sqlite3_vtab_rhs_value(sqlite3_index_info *, int, sqlite3_value * * ppVal);
int sqlite3_stmt_scanstatus(sqlite3_stmt * pStmt, int idx, int iScanStatusOp, void * pOut);
+int sqlite3_stmt_scanstatus_v2(sqlite3_stmt * pStmt, int idx, int iScanStatusOp, int flags, void * pOut);
void sqlite3_stmt_scanstatus_reset(sqlite3_stmt *);
int sqlite3_db_cacheflush(sqlite3 *);
int sqlite3_system_errno(sqlite3 *);
@@ -4566,6 +4605,8 @@
void sqlite3_snapshot_free(sqlite3_snapshot *);
int sqlite3_snapshot_cmp(sqlite3_snapshot * p1, sqlite3_snapshot * p2);
int sqlite3_snapshot_recover(sqlite3 * db, const char * zDb);
+unsigned char * sqlite3_serialize(sqlite3 * db, const char * zSchema, sqlite3_int64 * piSize, unsigned int mFlags);
+int sqlite3_deserialize(sqlite3 * db, const char * zSchema, unsigned char * pData, sqlite3_int64 szDb, sqlite3_int64 szBuf, unsigned mFlags);
typedef struct sqlite3_rtree_geometry sqlite3_rtree_geometry;
typedef struct sqlite3_rtree_query_info sqlite3_rtree_query_info;
typedef double sqlite3_rtree_dbl;
@@ -4625,8 +4666,19 @@
void(* xPhraseNext)(Fts5Context *, Fts5PhraseIter *, int * piCol, int * piOff);
int(* xPhraseFirstColumn)(Fts5Context *, int iPhrase, Fts5PhraseIter *, int *);
void(* xPhraseNextColumn)(Fts5Context *, Fts5PhraseIter *, int * piCol);
+ int(* xQueryToken)(Fts5Context *, int iPhrase, int iToken, const char * * ppToken, int * pnToken);
+ int(* xInstToken)(Fts5Context *, int iIdx, int iToken, const char * *, int *);
+ int(* xColumnLocale)(Fts5Context *, int iCol, const char * * pz, int * pn);
+ int(* xTokenize_v2)(Fts5Context *, const char * pText, int nText, const char * pLocale, int nLocale, void * pCtx, int(* xToken)(void *, int, const char *, int, int, int));
};
typedef struct Fts5Tokenizer Fts5Tokenizer;
+typedef struct fts5_tokenizer_v2 fts5_tokenizer_v2;
+struct fts5_tokenizer_v2 {
+ int iVersion;
+ int(* xCreate)(void *, const char * * azArg, int nArg, Fts5Tokenizer * * ppOut);
+ void(* xDelete)(Fts5Tokenizer *);
+ int(* xTokenize)(Fts5Tokenizer *, void * pCtx, int flags, const char * pText, int nText, const char * pLocale, int nLocale, int(* xToken)(void * pCtx, int tflags, const char * pToken, int nToken, int iStart, int iEnd));
+};
typedef struct fts5_tokenizer fts5_tokenizer;
struct fts5_tokenizer {
int(* xCreate)(void *, const char * * azArg, int nArg, Fts5Tokenizer * * ppOut);
@@ -4636,9 +4688,11 @@
typedef struct fts5_api fts5_api;
struct fts5_api {
int iVersion;
- int(* xCreateTokenizer)(fts5_api * pApi, const char * zName, void * pContext, fts5_tokenizer * pTokenizer, void(* xDestroy)(void *));
- int(* xFindTokenizer)(fts5_api * pApi, const char * zName, void * * ppContext, fts5_tokenizer * pTokenizer);
- int(* xCreateFunction)(fts5_api * pApi, const char * zName, void * pContext, fts5_extension_function xFunction, void(* xDestroy)(void *));
+ int(* xCreateTokenizer)(fts5_api * pApi, const char * zName, void * pUserData, fts5_tokenizer * pTokenizer, void(* xDestroy)(void *));
+ int(* xFindTokenizer)(fts5_api * pApi, const char * zName, void * * ppUserData, fts5_tokenizer * pTokenizer);
+ int(* xCreateFunction)(fts5_api * pApi, const char * zName, void * pUserData, fts5_extension_function xFunction, void(* xDestroy)(void *));
+ int(* xCreateTokenizer_v2)(fts5_api * pApi, const char * zName, void * pUserData, fts5_tokenizer_v2 * pTokenizer, void(* xDestroy)(void *));
+ int(* xFindTokenizer_v2)(fts5_api * pApi, const char * zName, void * * ppUserData, fts5_tokenizer_v2 * * ppTokenizer);
};
typedef sqlite3_int64 i64;
typedef sqlite3_uint64 u64;
@@ -7342,21 +7396,21 @@
int rc = 0;
(void)(pApi);
(void) pzErrMsg;
- rc = sqlite3_create_function(db, "sha3", 1, 1 | SQLITE_INNOCUOUS | 0x800, 0, sha3Func, 0, 0);
+ rc = sqlite3_create_function(db, "sha3", 1, 1 | 0x000200000 | 0x000000800, 0, sha3Func, 0, 0);
if(rc == 0) {
- rc = sqlite3_create_function(db, "sha3", 2, 1 | SQLITE_INNOCUOUS | 0x800, 0, sha3Func, 0, 0);
+ rc = sqlite3_create_function(db, "sha3", 2, 1 | 0x000200000 | 0x000000800, 0, sha3Func, 0, 0);
}
if(rc == 0) {
- rc = sqlite3_create_function(db, "sha3_agg", 1, 1 | SQLITE_INNOCUOUS | 0x800, 0, 0, sha3AggStep, sha3AggFinal);
+ rc = sqlite3_create_function(db, "sha3_agg", 1, 1 | 0x000200000 | 0x000000800, 0, 0, sha3AggStep, sha3AggFinal);
}
if(rc == 0) {
- rc = sqlite3_create_function(db, "sha3_agg", 2, 1 | SQLITE_INNOCUOUS | 0x800, 0, 0, sha3AggStep, sha3AggFinal);
+ rc = sqlite3_create_function(db, "sha3_agg", 2, 1 | 0x000200000 | 0x000000800, 0, 0, sha3AggStep, sha3AggFinal);
}
if(rc == 0) {
- rc = sqlite3_create_function(db, "sha3_query", 1, 1 | SQLITE_DIRECTONLY, 0, sha3QueryFunc, 0, 0);
+ rc = sqlite3_create_function(db, "sha3_query", 1, 1 | 0x000080000, 0, sha3QueryFunc, 0, 0);
}
if(rc == 0) {
- rc = sqlite3_create_function(db, "sha3_query", 2, 1 | SQLITE_DIRECTONLY, 0, sha3QueryFunc, 0, 0);
+ rc = sqlite3_create_function(db, "sha3_query", 2, 1 | 0x000080000, 0, sha3QueryFunc, 0, 0);
}
return rc;
}
@@ -7772,12 +7826,12 @@
static int one = 1;
(void)(pApi);
(void) pzErrMsg;
- rc = sqlite3_create_function(db, "sha1", 1, 1 | SQLITE_INNOCUOUS | 0x800, 0, sha1Func, 0, 0);
+ rc = sqlite3_create_function(db, "sha1", 1, 1 | 0x000200000 | 0x000000800, 0, sha1Func, 0, 0);
if(rc == 0) {
- rc = sqlite3_create_function(db, "sha1b", 1, 1 | SQLITE_INNOCUOUS | 0x800,(void *) & one, sha1Func, 0, 0);
+ rc = sqlite3_create_function(db, "sha1b", 1, 1 | 0x000200000 | 0x000000800,(void *) & one, sha1Func, 0, 0);
}
if(rc == 0) {
- rc = sqlite3_create_function(db, "sha1_query", 1, 1 | SQLITE_DIRECTONLY, 0, sha1QueryFunc, 0, 0);
+ rc = sqlite3_create_function(db, "sha1_query", 1, 1 | 0x000080000, 0, sha1QueryFunc, 0, 0);
}
return rc;
}
@@ -8502,10 +8556,10 @@
(void)(pApi);
for(i = 0;
i < (int)(sizeof(aFunc) / sizeof(aFunc[0])) && rc == 0; i ++ ) {
- rc = sqlite3_create_function(db, aFunc[i].zFuncName, aFunc[i].nArg, 1 | SQLITE_INNOCUOUS | 0x800, aFunc[i].iArg ? db: 0, aFunc[i].xFunc, 0, 0);
+ rc = sqlite3_create_function(db, aFunc[i].zFuncName, aFunc[i].nArg, 1 | 0x000200000 | 0x000000800, aFunc[i].iArg ? db: 0, aFunc[i].xFunc, 0, 0);
}
if(rc == 0) {
- rc = sqlite3_create_window_function(db, "decimal_sum", 1, 1 | SQLITE_INNOCUOUS | 0x800, 0, decimalSumStep, decimalSumFinalize, decimalSumValue, decimalSumInverse, 0);
+ rc = sqlite3_create_window_function(db, "decimal_sum", 1, 1 | 0x000200000 | 0x000000800, 0, decimalSumStep, decimalSumFinalize, decimalSumValue, decimalSumInverse, 0);
}
if(rc == 0) {
rc = sqlite3_create_collation(db, "decimal", 1, 0, decimalCollFunc);
@@ -8786,7 +8840,7 @@
(void) pzErrMsg;
for(i = 0;
i < sizeof(aPercentFunc) / sizeof(aPercentFunc[0]); i ++ ) {
- rc = sqlite3_create_window_function(db, aPercentFunc[i].zName, aPercentFunc[i].nArg, 1 | SQLITE_INNOCUOUS | SQLITE_SELFORDER1,(void *) & aPercentFunc[i], percentStep, percentFinal, percentValue, percentInverse, 0);
+ rc = sqlite3_create_window_function(db, aPercentFunc[i].zName, aPercentFunc[i].nArg, 1 | 0x000200000 | 0x002000000,(void *) & aPercentFunc[i], percentStep, percentFinal, percentValue, percentInverse, 0);
if(rc) break;
}
return rc;
@@ -8937,7 +8991,7 @@
int sqlite3_base64_init(sqlite3 * db, char * * pzErr, const sqlite3_api_routines * pApi) {
(void)(pApi);
(void) pzErr;
- return sqlite3_create_function(db, "base64", 1, 0x800 | SQLITE_INNOCUOUS | SQLITE_DIRECTONLY | 1, 0, base64, 0, 0);
+ return sqlite3_create_function(db, "base64", 1, 0x000000800 | 0x000200000 | 0x000080000 | 1, 0, base64, 0, 0);
};
static u8 b85_cOffset[] = { 0, '#', 0, '*' - 4, 0 };
static char * skipNonB85(char * s, int nc) {
@@ -9084,7 +9138,7 @@
int sqlite3_base85_init(sqlite3 * db, char * * pzErr, const sqlite3_api_routines * pApi) {
(void)(pApi);
(void) pzErr;
- return sqlite3_create_function(db, "base85", 1, 0x800 | SQLITE_INNOCUOUS | SQLITE_DIRECTONLY | 1, 0, base85, 0, 0);
+ return sqlite3_create_function(db, "base85", 1, 0x000000800 | 0x000200000 | 0x000080000 | 1, 0, base85, 0, 0);
}
static void ieee754func(sqlite3_context * context, int argc, sqlite3_value * * argv) {
if(argc == 1) {
@@ -9254,7 +9308,7 @@
(void) pzErrMsg;
for(i = 0;
i < sizeof(aFunc) / sizeof(aFunc[0]) && rc == 0; i ++ ) {
- rc = sqlite3_create_function(db, aFunc[i].zFName, aFunc[i].nArg, 1 | SQLITE_INNOCUOUS,(void *) & aFunc[i].iAux, aFunc[i].xFunc, 0, 0);
+ rc = sqlite3_create_function(db, aFunc[i].zFName, aFunc[i].nArg, 1 | 0x000200000,(void *) & aFunc[i].iAux, aFunc[i].xFunc, 0, 0);
}
return rc;
}
@@ -9371,7 +9425,7 @@
pNew = * ppVtab = sqlite3_malloc(sizeof( * pNew));
if(pNew == 0) return 7;
memset(pNew, 0, sizeof( * pNew));
- sqlite3_vtab_config(db, SQLITE_VTAB_INNOCUOUS);
+ sqlite3_vtab_config(db, 2);
}
return rc;
}
@@ -9611,15 +9665,15 @@
int iCol;
int iMask;
int op = pConstraint->op;
- if(op >= SQLITE_INDEX_CONSTRAINT_LIMIT && op <= SQLITE_INDEX_CONSTRAINT_OFFSET) {
+ if(op >= 73 && op <= 74) {
if(pConstraint->usable == 0) {
}
- else if(op == SQLITE_INDEX_CONSTRAINT_LIMIT) {
+ else if(op == 73) {
aIdx[3] = i;
idxNum |= 0x20;
}
else {
- ((op == SQLITE_INDEX_CONSTRAINT_OFFSET) ?(void)(0): __assert_fail("op==SQLITE_INDEX_CONSTRAINT_OFFSET", "fossil0/extsrc/shell.c", 6703, __PRETTY_FUNCTION__));
+ ((op == 74) ?(void)(0): __assert_fail("op==SQLITE_INDEX_CONSTRAINT_OFFSET", "fossil0/extsrc/shell.c", 6703, __PRETTY_FUNCTION__));
aIdx[4] = i;
idxNum |= 0x40;
}
@@ -9730,6 +9784,7 @@
pIdxInfo->estimatedRows = 2147483647;
}
pIdxInfo->idxNum = idxNum;
+ pIdxInfo->idxFlags = 0x00000002;
return 0;
}
static sqlite3_module seriesModule = { 0, 0, seriesConnect, seriesBestIndex, seriesDisconnect, 0, seriesOpen, seriesClose, seriesFilter, seriesNext, seriesEof, seriesColumn, seriesRowid, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 };
@@ -10370,9 +10425,9 @@
int rc = 0;
(void)(pApi);
(void) pzErrMsg;
- rc = sqlite3_create_function(db, "regexp", 2, 1 | SQLITE_INNOCUOUS | 0x800, 0, re_sql_func, 0, 0);
+ rc = sqlite3_create_function(db, "regexp", 2, 1 | 0x000200000 | 0x000000800, 0, re_sql_func, 0, 0);
if(rc == 0) {
- rc = sqlite3_create_function(db, "regexpi", 2, 1 | SQLITE_INNOCUOUS | 0x800,(void *) db, re_sql_func, 0, 0);
+ rc = sqlite3_create_function(db, "regexpi", 2, 1 | 0x000200000 | 0x000000800,(void *) db, re_sql_func, 0, 0);
}
return rc;
}
@@ -10817,7 +10872,7 @@
pNew = (fsdir_tab *) sqlite3_malloc(sizeof( * pNew));
if(pNew == 0) return 7;
memset(pNew, 0, sizeof( * pNew));
- sqlite3_vtab_config(db, SQLITE_VTAB_DIRECTONLY);
+ sqlite3_vtab_config(db, 3);
}
* ppVtab = (sqlite3_vtab *) pNew;
return rc;
@@ -11127,9 +11182,9 @@
int rc = 0;
(void)(pApi);
(void) pzErrMsg;
- rc = sqlite3_create_function(db, "readfile", 1, 1 | SQLITE_DIRECTONLY, 0, readfileFunc, 0, 0);
+ rc = sqlite3_create_function(db, "readfile", 1, 1 | 0x000080000, 0, readfileFunc, 0, 0);
if(rc == 0) {
- rc = sqlite3_create_function(db, "writefile", - 1, 1 | SQLITE_DIRECTONLY, 0, writefileFunc, 0, 0);
+ rc = sqlite3_create_function(db, "writefile", - 1, 1 | 0x000080000, 0, writefileFunc, 0, 0);
}
if(rc == 0) {
rc = sqlite3_create_function(db, "lsmode", 1, 1, 0, lsModeFunc, 0, 0);
@@ -11165,7 +11220,7 @@
(void)(argc);
(void)(argv);
(void)(pzErr);
- sqlite3_vtab_config(db, SQLITE_VTAB_INNOCUOUS);
+ sqlite3_vtab_config(db, 2);
rc = sqlite3_declare_vtab(db, "CREATE TABLE x( candidate TEXT, prefix TEXT HIDDEN, wholeline TEXT HIDDEN, phase INT HIDDEN)");
if(rc == 0) {
pNew = sqlite3_malloc(sizeof( * pNew));
@@ -12368,7 +12423,7 @@
}
while(collisions < 50 && zName != 0);
if(collisions) {
- rc = SQLITE_BUSY_TIMEOUT;
+ rc = (5 | (3 << 8));
}
else if(zName == 0) {
rc = 7;
@@ -12387,7 +12442,7 @@
else {
rc = sqlite3_exec(dbm, zIdx, 0, 0, p->pzErrmsg);
if(rc != 0) {
- rc = SQLITE_BUSY_TIMEOUT;
+ rc = (5 | (3 << 8));
}
else {
idxHashAdd( & rc, & p->hIdx, zName, zIdx);
@@ -13064,7 +13119,7 @@
int rcf = 1;
if(strcmp(enc, "utf16le") == 0) ienc = 2;
else if(strcmp(enc, "utf16be") == 0) ienc = 3;
- ienc |= (flags & (0x800 | SQLITE_DIRECTONLY));
+ ienc |= (flags & (0x000000800 | 0x000080000));
if(strcmp(type, "w") == 0) {
rcf = sqlite3_create_window_function(dbDst, name, nargs, ienc, 0, dummyUDF, dummyUDFvalue, 0, 0, 0);
}
@@ -13200,7 +13255,7 @@
if(rc == 0) {
rc = idxCreateCandidates(p);
}
- else if(rc == SQLITE_BUSY_TIMEOUT) {
+ else if(rc == (5 | (3 << 8))) {
if(pzErr) * pzErr = sqlite3_mprintf("Cannot find a unique index name to propose.");
return rc;
}
@@ -14281,34 +14336,34 @@
zOp = "ROLLBACK_ATOMIC_WRITE";
break;
}
- case SQLITE_FCNTL_LOCK_TIMEOUT:
+ case 34:
{
sqlite3_snprintf(sizeof(zBuf), zBuf, "LOCK_TIMEOUT,%d", * (int *) pArg);
zOp = zBuf;
break;
}
- case SQLITE_FCNTL_DATA_VERSION:
+ case 35:
zOp = "DATA_VERSION";
break;
- case SQLITE_FCNTL_SIZE_LIMIT:
+ case 36:
zOp = "SIZE_LIMIT";
break;
- case SQLITE_FCNTL_CKPT_DONE:
+ case 37:
zOp = "CKPT_DONE";
break;
- case SQLITE_FCNTL_RESERVE_BYTES:
+ case 38:
zOp = "RESERVED_BYTES";
break;
- case SQLITE_FCNTL_CKPT_START:
+ case 39:
zOp = "CKPT_START";
break;
- case SQLITE_FCNTL_EXTERNAL_READER:
+ case 40:
zOp = "EXTERNAL_READER";
break;
- case SQLITE_FCNTL_CKSM_FILE:
+ case 41:
zOp = "CKSM_FILE";
break;
- case SQLITE_FCNTL_RESET_CACHE:
+ case 42:
zOp = "RESET_CACHE";
break;
case 0xca093fa0:
@@ -16211,7 +16266,7 @@
sqlite3_db_status(db, 9, & iCur, & iHiwtr, 1);
fprintf(out, "Page cache writes: %d\n", iCur);
iHiwtr = iCur = - 1;
- sqlite3_db_status(db, SQLITE_DBSTATUS_CACHE_SPILL, & iCur, & iHiwtr, 1);
+ sqlite3_db_status(db, 12, & iCur, & iHiwtr, 1);
fprintf(out, "Page cache spills: %d\n", iCur);
iHiwtr = iCur = - 1;
sqlite3_db_status(db, 2, & iCur, & iHiwtr, bReset);
@@ -16228,8 +16283,8 @@
fprintf(out, "Sort Operations: %d\n", iCur);
iCur = sqlite3_stmt_status(pArg->pStmt, 3, bReset);
fprintf(out, "Autoindex Inserts: %d\n", iCur);
- iHit = sqlite3_stmt_status(pArg->pStmt, SQLITE_STMTSTATUS_FILTER_HIT, bReset);
- iMiss = sqlite3_stmt_status(pArg->pStmt, SQLITE_STMTSTATUS_FILTER_MISS, bReset);
+ iHit = sqlite3_stmt_status(pArg->pStmt, 8, bReset);
+ iMiss = sqlite3_stmt_status(pArg->pStmt, 7, bReset);
if(iHit || iMiss) {
fprintf(out, "Bloom filter bypass taken: %d/%d\n", iHit, iHit + iMiss);
}
@@ -16311,25 +16366,25 @@
static unsigned int savedWhereTrace;
static void disable_debug_trace_modes(void) {
unsigned int zero = 0;
- sqlite3_test_control(SQLITE_TESTCTRL_TRACEFLAGS, 0, & savedSelectTrace);
- sqlite3_test_control(SQLITE_TESTCTRL_TRACEFLAGS, 1, & zero);
- sqlite3_test_control(SQLITE_TESTCTRL_TRACEFLAGS, 2, & savedWhereTrace);
- sqlite3_test_control(SQLITE_TESTCTRL_TRACEFLAGS, 3, & zero);
+ sqlite3_test_control(31, 0, & savedSelectTrace);
+ sqlite3_test_control(31, 1, & zero);
+ sqlite3_test_control(31, 2, & savedWhereTrace);
+ sqlite3_test_control(31, 3, & zero);
}
static void restore_debug_trace_modes(void) {
- sqlite3_test_control(SQLITE_TESTCTRL_TRACEFLAGS, 1, & savedSelectTrace);
- sqlite3_test_control(SQLITE_TESTCTRL_TRACEFLAGS, 3, & savedWhereTrace);
+ sqlite3_test_control(31, 1, & savedSelectTrace);
+ sqlite3_test_control(31, 3, & savedWhereTrace);
}
static void bind_table_init(ShellState * p) {
int wrSchema = 0;
int defensiveMode = 0;
- sqlite3_db_config(p->db, SQLITE_DBCONFIG_DEFENSIVE, - 1, & defensiveMode);
- sqlite3_db_config(p->db, SQLITE_DBCONFIG_DEFENSIVE, 0, 0);
- sqlite3_db_config(p->db, SQLITE_DBCONFIG_WRITABLE_SCHEMA, - 1, & wrSchema);
- sqlite3_db_config(p->db, SQLITE_DBCONFIG_WRITABLE_SCHEMA, 1, 0);
+ sqlite3_db_config(p->db, 1010, - 1, & defensiveMode);
+ sqlite3_db_config(p->db, 1010, 0, 0);
+ sqlite3_db_config(p->db, 1011, - 1, & wrSchema);
+ sqlite3_db_config(p->db, 1011, 1, 0);
sqlite3_exec(p->db, "CREATE TABLE IF NOT EXISTS temp.sqlite_parameters(\n key TEXT PRIMARY KEY,\n value\n) WITHOUT ROWID;", 0, 0, 0);
- sqlite3_db_config(p->db, SQLITE_DBCONFIG_WRITABLE_SCHEMA, wrSchema, 0);
- sqlite3_db_config(p->db, SQLITE_DBCONFIG_DEFENSIVE, defensiveMode, 0);
+ sqlite3_db_config(p->db, 1011, wrSchema, 0);
+ sqlite3_db_config(p->db, 1010, defensiveMode, 0);
}
static void bind_prepared_stmt(ShellState * pArg, sqlite3_stmt * pStmt) {
int nVar;
@@ -17614,11 +17669,11 @@
}
}
globalDb = p->db;
- sqlite3_db_config(p->db, SQLITE_DBCONFIG_STMT_SCANSTATUS,(int) 0,(int *) 0);
+ sqlite3_db_config(p->db, 1018,(int) 0,(int *) 0);
{
int testmode_on = (((p)->shellFlgs & (0x00000400)) != 0);
- sqlite3_db_config(p->db, SQLITE_DBCONFIG_TRUSTED_SCHEMA, testmode_on, 0);
- sqlite3_db_config(p->db, SQLITE_DBCONFIG_DEFENSIVE, ! testmode_on, 0);
+ sqlite3_db_config(p->db, 1017, testmode_on, 0);
+ sqlite3_db_config(p->db, 1010, ! testmode_on, 0);
}
sqlite3_enable_load_extension(p->db, 1);
sqlite3_sha_init(p->db, 0, 0);
@@ -17662,12 +17717,12 @@
if(aData == 0) {
return;
}
- rc = sqlite3_deserialize(p->db, "main", aData, nData, nData, SQLITE_DESERIALIZE_RESIZEABLE | SQLITE_DESERIALIZE_FREEONCLOSE);
+ rc = sqlite3_deserialize(p->db, "main", aData, nData, nData, 2 | 1);
if(rc) {
fprintf(stderr, "Error: sqlite3_deserialize() returns %d\n", rc);
}
if(p->szMax > 0) {
- sqlite3_file_control(p->db, "main", SQLITE_FCNTL_SIZE_LIMIT, & p->szMax);
+ sqlite3_file_control(p->db, "main", 36, & p->szMax);
}
}
}
@@ -17675,7 +17730,7 @@
if(p->bSafeModePersist) {
sqlite3_set_authorizer(p->db, safeModeAuth, p);
}
- sqlite3_db_config(p->db, SQLITE_DBCONFIG_STMT_SCANSTATUS, p->scanstatsOn,(int *) 0);
+ sqlite3_db_config(p->db, 1018, p->scanstatsOn,(int *) 0);
}
}
void close_db(sqlite3 * db) {
@@ -19069,7 +19124,7 @@
int eTxn = sqlite3_txn_state(p->db, azName[i * 2]);
int bRdonly = sqlite3_db_readonly(p->db, azName[i * 2]);
const char * z = azName[i * 2 + 1];
- fprintf(p->out, "%s: %s %s%s\n", azName[i * 2], z && z[0] ? z: "\"\"", bRdonly ? "r/o": "r/w", eTxn == SQLITE_TXN_NONE ? "": eTxn == SQLITE_TXN_READ ? " read-txn": " write-txn");
+ fprintf(p->out, "%s: %s %s%s\n", azName[i * 2], z && z[0] ? z: "\"\"", bRdonly ? "r/o": "r/w", eTxn == 0 ? "": eTxn == 1 ? " read-txn": " write-txn");
free(azName[i * 2]);
free(azName[i * 2 + 1]);
}
@@ -19080,7 +19135,7 @@
const char * zName;
int op;
}
- aDbConfig[] = { { "attach_create", SQLITE_DBCONFIG_ENABLE_ATTACH_CREATE }, { "attach_write", SQLITE_DBCONFIG_ENABLE_ATTACH_WRITE }, { "comments", SQLITE_DBCONFIG_ENABLE_COMMENTS }, { "defensive", SQLITE_DBCONFIG_DEFENSIVE }, { "dqs_ddl", SQLITE_DBCONFIG_DQS_DDL }, { "dqs_dml", SQLITE_DBCONFIG_DQS_DML }, { "enable_fkey", 1002 }, { "enable_qpsg", 1007 }, { "enable_trigger", 1003 }, { "enable_view", SQLITE_DBCONFIG_ENABLE_VIEW }, { "fts3_tokenizer", 1004 }, { "legacy_alter_table", SQLITE_DBCONFIG_LEGACY_ALTER_TABLE }, { "legacy_file_format", SQLITE_DBCONFIG_LEGACY_FILE_FORMAT }, { "load_extension", 1005 }, { "no_ckpt_on_close", 1006 }, { "reset_database", SQLITE_DBCONFIG_RESET_DATABASE }, { "reverse_scanorder", SQLITE_DBCONFIG_REVERSE_SCANORDER }, { "stmt_scanstatus", SQLITE_DBCONFIG_STMT_SCANSTATUS }, { "trigger_eqp", 1008 }, { "trusted_schema", SQLITE_DBCONFIG_TRUSTED_SCHEMA }, { "writable_schema", SQLITE_DBCONFIG_WRITABLE_SCHEMA } };
+ aDbConfig[] = { { "attach_create", 1020 }, { "attach_write", 1021 }, { "comments", 1022 }, { "defensive", 1010 }, { "dqs_ddl", 1014 }, { "dqs_dml", 1013 }, { "enable_fkey", 1002 }, { "enable_qpsg", 1007 }, { "enable_trigger", 1003 }, { "enable_view", 1015 }, { "fts3_tokenizer", 1004 }, { "legacy_alter_table", 1012 }, { "legacy_file_format", 1016 }, { "load_extension", 1005 }, { "no_ckpt_on_close", 1006 }, { "reset_database", 1009 }, { "reverse_scanorder", 1019 }, { "stmt_scanstatus", 1018 }, { "trigger_eqp", 1008 }, { "trusted_schema", 1017 }, { "writable_schema", 1011 } };
int ii, v;
open_db(p, 0);
for(ii = 0;
@@ -19249,7 +19304,7 @@
int ctrlCode;
const char * zUsage;
}
- aCtrl[] = { { "chunk_size", 6, "SIZE" }, { "data_version", SQLITE_FCNTL_DATA_VERSION, "" }, { "has_moved", 20, "" }, { "lock_timeout", SQLITE_FCNTL_LOCK_TIMEOUT, "MILLISEC" }, { "persist_wal", 10, "[BOOLEAN]" }, { "psow", 13, "[BOOLEAN]" }, { "reserve_bytes", SQLITE_FCNTL_RESERVE_BYTES, "[N]" }, { "size_limit", SQLITE_FCNTL_SIZE_LIMIT, "[LIMIT]" }, { "tempfilename", 16, "" } };
+ aCtrl[] = { { "chunk_size", 6, "SIZE" }, { "data_version", 35, "" }, { "has_moved", 20, "" }, { "lock_timeout", 34, "MILLISEC" }, { "persist_wal", 10, "[BOOLEAN]" }, { "psow", 13, "[BOOLEAN]" }, { "reserve_bytes", 38, "[N]" }, { "size_limit", 36, "[LIMIT]" }, { "tempfilename", 16, "" } };
int filectrl = - 1;
int iCtrl = - 1;
sqlite3_int64 iRes = 0;
@@ -19299,15 +19354,15 @@
}
else {
switch(filectrl) {
- case SQLITE_FCNTL_SIZE_LIMIT:
+ case 36:
{
if(nArg != 2 && nArg != 3) break;
iRes = nArg == 3 ? integerValue(azArg[2]): - 1;
- sqlite3_file_control(p->db, zSchema, SQLITE_FCNTL_SIZE_LIMIT, & iRes);
+ sqlite3_file_control(p->db, zSchema, 36, & iRes);
isOk = 1;
break;
}
- case SQLITE_FCNTL_LOCK_TIMEOUT:
+ case 34:
case 6:
{
int x;
@@ -19328,7 +19383,7 @@
isOk = 1;
break;
}
- case SQLITE_FCNTL_DATA_VERSION:
+ case 35:
case 20:
{
int x;
@@ -19350,7 +19405,7 @@
isOk = 2;
break;
}
- case SQLITE_FCNTL_RESERVE_BYTES:
+ case 38:
{
int x;
if(nArg >= 3) {
@@ -20119,7 +20174,7 @@
openMode = 4;
}
else if(optionMatch(z, "nofollow")) {
- p->openFlags |= SQLITE_OPEN_NOFOLLOW;
+ p->openFlags |= 0x01000000;
}
else if(optionMatch(z, "deserialize")) {
openMode = 5;
@@ -20540,7 +20595,7 @@
p->scanstatsOn = (u8) booleanValue(azArg[1]);
}
open_db(p, 0);
- sqlite3_db_config(p->db, SQLITE_DBCONFIG_STMT_SCANSTATUS, p->scanstatsOn,(int *) 0);
+ sqlite3_db_config(p->db, 1018, p->scanstatsOn,(int *) 0);
fputs("Warning: .scanstats not available in this build.\n", stderr);
}
else {
@@ -20684,7 +20739,7 @@
}
else if((c == 's' && n == 11 && cli_strncmp(azArg[0], "selecttrace", n) == 0) || (c == 't' && n == 9 && cli_strncmp(azArg[0], "treetrace", n) == 0)) {
unsigned int x = nArg >= 2 ?(unsigned int) integerValue(azArg[1]): 0xffffffff;
- sqlite3_test_control(SQLITE_TESTCTRL_TRACEFLAGS, 1, & x);
+ sqlite3_test_control(31, 1, & x);
}
else if(c == 's' && n >= 4 && cli_strncmp(azArg[0], "selftest", n) == 0) {
int bIsInit = 0;
@@ -21150,7 +21205,7 @@
int unSafe;
const char * zUsage;
}
- aCtrl[] = { { "always", 13, 1, "BOOLEAN" }, { "assert", 12, 1, "BOOLEAN" }, { "bitvec_test", 8, 1, "SIZE INT-ARRAY" }, { "byteorder", 22, 0, "" }, { "extra_schema_checks", SQLITE_TESTCTRL_EXTRA_SCHEMA_CHECKS, 0, "BOOLEAN" }, { "fault_install", 9, 1, "args..." }, { "fk_no_action", SQLITE_TESTCTRL_FK_NO_ACTION, 0, "BOOLEAN" }, { "imposter", 25, 1, "SCHEMA ON/OFF ROOTPAGE" }, { "internal_functions", SQLITE_TESTCTRL_INTERNAL_FUNCTIONS, 0, "" }, { "json_selfcheck", SQLITE_TESTCTRL_JSON_SELFCHECK, 0, "BOOLEAN" }, { "localtime_fault", 18, 0, "BOOLEAN" }, { "never_corrupt", 20, 1, "BOOLEAN" }, { "optimizations", 15, 0, "DISABLE-MASK ..." }, { "pending_byte", 11, 1, "OFFSET " }, { "prng_restore", 6, 0, "" }, { "prng_save", 5, 0, "" }, { "prng_seed", SQLITE_TESTCTRL_PRNG_SEED, 0, "SEED ?db?" }, { "seek_count", SQLITE_TESTCTRL_SEEK_COUNT, 0, "" }, { "sorter_mmap", 24, 0, "NMAX" }, { "tune", SQLITE_TESTCTRL_TUNE, 1, "ID VALUE" } };
+ aCtrl[] = { { "always", 13, 1, "BOOLEAN" }, { "assert", 12, 1, "BOOLEAN" }, { "bitvec_test", 8, 1, "SIZE INT-ARRAY" }, { "byteorder", 22, 0, "" }, { "extra_schema_checks", 29, 0, "BOOLEAN" }, { "fault_install", 9, 1, "args..." }, { "fk_no_action", 7, 0, "BOOLEAN" }, { "imposter", 25, 1, "SCHEMA ON/OFF ROOTPAGE" }, { "internal_functions", 17, 0, "" }, { "json_selfcheck", 14, 0, "BOOLEAN" }, { "localtime_fault", 18, 0, "BOOLEAN" }, { "never_corrupt", 20, 1, "BOOLEAN" }, { "optimizations", 15, 0, "DISABLE-MASK ..." }, { "pending_byte", 11, 1, "OFFSET " }, { "prng_restore", 6, 0, "" }, { "prng_save", 5, 0, "" }, { "prng_seed", 28, 0, "SEED ?db?" }, { "seek_count", 30, 0, "" }, { "sorter_mmap", 24, 0, "NMAX" }, { "tune", 32, 1, "ID VALUE" } };
int testctrl = - 1;
int iCtrl = - 1;
int rc2 = 0;
@@ -21207,7 +21262,7 @@
unsigned int m;
int ii;
int nOff;
- sqlite3_test_control(SQLITE_TESTCTRL_GETOPT, p->db, & curOpt);
+ sqlite3_test_control(16, p->db, & curOpt);
newOpt = curOpt;
for(ii = 2;
ii < nArg; ii ++ ) {
@@ -21281,7 +21336,7 @@
rc2 = isOk = 3;
break;
}
- case SQLITE_TESTCTRL_FK_NO_ACTION:
+ case 7:
if(nArg == 3) {
unsigned int opt = (unsigned int) strtol(azArg[2], 0, 0);
rc2 = sqlite3_test_control(testctrl, p->db, opt);
@@ -21303,7 +21358,7 @@
isOk = 3;
}
break;
- case SQLITE_TESTCTRL_PRNG_SEED:
+ case 28:
if(nArg == 3 || nArg == 4) {
int ii = (int) integerValue(azArg[2]);
sqlite3 * db;
@@ -21338,7 +21393,7 @@
isOk = 3;
}
break;
- case SQLITE_TESTCTRL_INTERNAL_FUNCTIONS:
+ case 17:
rc2 = sqlite3_test_control(testctrl, p->db);
isOk = 3;
break;
@@ -21348,7 +21403,7 @@
isOk = 3;
}
break;
- case SQLITE_TESTCTRL_SEEK_COUNT:
+ case 30:
{
u64 x = 0;
rc2 = sqlite3_test_control(testctrl, p->db, & x);
@@ -21363,7 +21418,7 @@
isOk = 3;
}
break;
- case SQLITE_TESTCTRL_JSON_SELFCHECK:
+ case 14:
if(nArg == 2) {
rc2 = - 1;
isOk = 1;
@@ -21601,7 +21656,7 @@
}
else if(c == 'w' && cli_strncmp(azArg[0], "wheretrace", n) == 0) {
unsigned int x = nArg >= 2 ?(unsigned int) integerValue(azArg[1]): 0xffffffff;
- sqlite3_test_control(SQLITE_TESTCTRL_TRACEFLAGS, 3, & x);
+ sqlite3_test_control(31, 3, & x);
}
else if(c == 'w' && cli_strncmp(azArg[0], "width", n) == 0) {
int j;
@@ -21760,10 +21815,10 @@
if(bIsDump && rc == 0) {
int bDefense = 0;
int bDqsDdl = 0;
- sqlite3_db_config(p->db, SQLITE_DBCONFIG_DEFENSIVE, - 1, & bDefense);
- sqlite3_db_config(p->db, SQLITE_DBCONFIG_DQS_DDL, - 1, & bDqsDdl);
- sqlite3_db_config(p->db, SQLITE_DBCONFIG_DEFENSIVE, 0, 0);
- sqlite3_db_config(p->db, SQLITE_DBCONFIG_DQS_DDL, 1, 0);
+ sqlite3_db_config(p->db, 1010, - 1, & bDefense);
+ sqlite3_db_config(p->db, 1014, - 1, & bDqsDdl);
+ sqlite3_db_config(p->db, 1010, 0, 0);
+ sqlite3_db_config(p->db, 1014, 1, 0);
p->eRestoreState = (bDefense ? 2: 0) + (bDqsDdl ? 4: 0);
}
else {
@@ -21774,10 +21829,10 @@
default: {
if(sqlite3_get_autocommit(p->db)) {
if((p->eRestoreState & 2)) {
- sqlite3_db_config(p->db, SQLITE_DBCONFIG_DEFENSIVE, 1, 0);
+ sqlite3_db_config(p->db, 1010, 1, 0);
}
if((p->eRestoreState & 4)) {
- sqlite3_db_config(p->db, SQLITE_DBCONFIG_DQS_DDL, 0, 0);
+ sqlite3_db_config(p->db, 1014, 0, 0);
}
p->eRestoreState = 7;
}
@@ -22123,8 +22178,8 @@
}
}
signal(2, interrupt_handler);
- if(cli_strncmp(sqlite3_sourceid(), "2018-01-22 18:45:57 0c55d179733b46d8d0ba4d88e01a25e10677046ee3da1d5b1581e86726f2alt1", 60) != 0) {
- fprintf(stderr, "SQLite header and source version mismatch\n%s\n%s\n", sqlite3_sourceid(), "2018-01-22 18:45:57 0c55d179733b46d8d0ba4d88e01a25e10677046ee3da1d5b1581e86726f2alt1");
+ if(cli_strncmp(sqlite3_sourceid(), "2025-06-27 19:02:21 5508b56fd24016c13981ec280ecdd833007c9d8dd595edb295b984c2b487b5c8", 60) != 0) {
+ fprintf(stderr, "SQLite header and source version mismatch\n%s\n%s\n", sqlite3_sourceid(), "2025-06-27 19:02:21 5508b56fd24016c13981ec280ecdd833007c9d8dd595edb295b984c2b487b5c8");
exit(1);
}
main_init(& data);
@@ -22170,7 +22225,7 @@
}
else if(cli_strcmp(z, "-no-rowid-in-view") == 0) {
int val = 0;
- sqlite3_config(SQLITE_CONFIG_ROWID_IN_VIEW, & val);
+ sqlite3_config(30, & val);
((val == 0) ?(void)(0): __assert_fail("val==0", "fossil0/extsrc/shell.c", 33328, __PRETTY_FUNCTION__));
}
else if(cli_strcmp(z, "-heap") == 0) {
@@ -22238,7 +22293,7 @@
data.openMode = 4;
}
else if(cli_strcmp(z, "-nofollow") == 0) {
- data.openFlags = SQLITE_OPEN_NOFOLLOW;
+ data.openFlags = 0x01000000;
}
else if(cli_strcmp(z, "-memtrace") == 0) {
sqlite3MemTraceActivate(stderr);
@@ -22363,7 +22418,7 @@
data.openMode = 4;
}
else if(cli_strcmp(z, "-nofollow") == 0) {
- data.openFlags |= SQLITE_OPEN_NOFOLLOW;
+ data.openFlags |= 0x01000000;
}
else if(cli_strcmp(z, "-ascii") == 0) {
data.mode = 10;
@@ -22583,6 +22638,6 @@
return rc;
}
/*
-0.46user 0.03system 0:00.50elapsed 97%CPU (0avgtext+0avgdata 57712maxresident)k
-3160inputs+0outputs (0major+21419minor)pagefaults 0swaps
+0.46user 0.04system 0:00.51elapsed 99%CPU (0avgtext+0avgdata 58000maxresident)k
+0inputs+0outputs (0major+21520minor)pagefaults 0swaps
*/
It seems like cxx doesn't search by default the source folder for includes, gcc/clang seems to do it.
It seems that there is some advantage in having cxx been able to ignore non declared enum/functions/struct with a custom ASTPrinter to convert C/C++ code to another programming language because then we get named constants instead of raw numbers.
Also looking at the GCCLinuxToolchain.cpp the macro defineMacro("__cplusplus", "202400L"); is defined twice.
Thanks, @mingodad. I’ll take a look at your changes this weekend. Getting C to work is quite challenging; it’s a lot of work. There is a script in the repository to regenerate the toolchains e.g. https://github.com/robertoraggi/cplusplus/blob/main/scripts/update-predefined-macros.mjs but I have an improved version that relies on docker, I will clean it up and push it.
Thank you for reply !
As you'll see I've started with basically not defining any C++ macro and not including C++ include path to no have namespace/extern "C"/using xxxx in the output and checked for some C++ keywords and return then as identifiers (right now only wchar_t and new to be able to parse sqlite3.c).
It seems that paying attention to class members memory alignment can shave a bit of memory usage (see bellow a program that shows the sizeof of all AST nodes), for example for sizeof(cxx::TemplateDeclarationAST) = 64 if we reorder it's members then sizeof(cxx::MyTemplateDeclarationAST) = 56 (see bellow).
/*
zig c++ -std=c++23 -I../src/parser ast-sizeof.cpp
*/
#include <stdio.h>
#include "cxx/ast.h"
namespace cxx {
class MyAST : public Managed {
public:
virtual void accept(ASTVisitor* visitor) {};
ASTKind kind_;
};
class MyDeclarationAST : public MyAST {
public:
};
class MyTemplateDeclarationAST final : public MyDeclarationAST {
public:
SourceLocation templateLoc;
SourceLocation lessLoc;
SourceLocation greaterLoc;
List<TemplateParameterAST*>* templateParameterList = nullptr;
RequiresClauseAST* requiresClause = nullptr;
DeclarationAST* declaration = nullptr;
TemplateParametersSymbol* symbol = nullptr;
};
class MyExpressionAST : public MyAST {
public:
ValueCategory valueCategory = ValueCategory::kPrValue;
const Type* type = nullptr;
};
class MyMemberExpressionAST final : public MyExpressionAST {
public:
SourceLocation accessLoc;
SourceLocation templateLoc;
ExpressionAST* baseExpression = nullptr;
NestedNameSpecifierAST* nestedNameSpecifier = nullptr;
UnqualifiedIdAST* unqualifiedId = nullptr;
Symbol* symbol = nullptr;
TokenKind accessOp = TokenKind::T_EOF_SYMBOL;
bool isTemplateIntroduced = false;
};
class MyMemberExpressionAST2 : public ExpressionAST {
public:
static constexpr ASTKind Kind = ASTKind::MemberExpression;
MyMemberExpressionAST2(): ExpressionAST(Kind) {}
SourceLocation accessLoc;
SourceLocation templateLoc;
ExpressionAST* baseExpression = nullptr;
NestedNameSpecifierAST* nestedNameSpecifier = nullptr;
UnqualifiedIdAST* unqualifiedId = nullptr;
Symbol* symbol = nullptr;
TokenKind accessOp = TokenKind::T_EOF_SYMBOL;
bool isTemplateIntroduced = false;
//void accept(ASTVisitor* visitor) override { }
//auto firstSourceLocation() -> SourceLocation override;
//auto lastSourceLocation() -> SourceLocation override;
};
class MySpecifierAST : public MyAST {
public:
};
class MyVaListTypeSpecifierAST final : public MySpecifierAST {
public:
TokenKind specifier = TokenKind::T_EOF_SYMBOL;
SourceLocation specifierLoc;
};
}
int main(int argc, char *argv[]) {
// UnitAST
printf("sizeof(TranslationUnitAST) = %zd\n", sizeof(cxx::TranslationUnitAST));
printf("sizeof(ModuleUnitAST) = %zd\n", sizeof(cxx::ModuleUnitAST));
// DeclarationAST
printf("sizeof(SimpleDeclarationAST) = %zd\n", sizeof(cxx::SimpleDeclarationAST));
printf("sizeof(AsmDeclarationAST) = %zd\n", sizeof(cxx::AsmDeclarationAST));
printf("sizeof(NamespaceAliasDefinitionAST) = %zd\n", sizeof(cxx::NamespaceAliasDefinitionAST));
printf("sizeof(UsingDeclarationAST) = %zd\n", sizeof(cxx::UsingDeclarationAST));
printf("sizeof(UsingEnumDeclarationAST) = %zd\n", sizeof(cxx::UsingEnumDeclarationAST));
printf("sizeof(UsingDirectiveAST) = %zd\n", sizeof(cxx::UsingDirectiveAST));
printf("sizeof(StaticAssertDeclarationAST) = %zd\n", sizeof(cxx::StaticAssertDeclarationAST));
printf("sizeof(AliasDeclarationAST) = %zd\n", sizeof(cxx::AliasDeclarationAST));
printf("sizeof(OpaqueEnumDeclarationAST) = %zd\n", sizeof(cxx::OpaqueEnumDeclarationAST));
printf("sizeof(FunctionDefinitionAST) = %zd\n", sizeof(cxx::FunctionDefinitionAST));
printf("sizeof(TemplateDeclarationAST) = %zd\n", sizeof(cxx::TemplateDeclarationAST));
printf("sizeof(ConceptDefinitionAST) = %zd\n", sizeof(cxx::ConceptDefinitionAST));
printf("sizeof(DeductionGuideAST) = %zd\n", sizeof(cxx::DeductionGuideAST));
printf("sizeof(ExplicitInstantiationAST) = %zd\n", sizeof(cxx::ExplicitInstantiationAST));
printf("sizeof(ExportDeclarationAST) = %zd\n", sizeof(cxx::ExportDeclarationAST));
printf("sizeof(ExportCompoundDeclarationAST) = %zd\n", sizeof(cxx::ExportCompoundDeclarationAST));
printf("sizeof(LinkageSpecificationAST) = %zd\n", sizeof(cxx::LinkageSpecificationAST));
printf("sizeof(NamespaceDefinitionAST) = %zd\n", sizeof(cxx::NamespaceDefinitionAST));
printf("sizeof(EmptyDeclarationAST) = %zd\n", sizeof(cxx::EmptyDeclarationAST));
printf("sizeof(AttributeDeclarationAST) = %zd\n", sizeof(cxx::AttributeDeclarationAST));
printf("sizeof(ModuleImportDeclarationAST) = %zd\n", sizeof(cxx::ModuleImportDeclarationAST));
printf("sizeof(ParameterDeclarationAST) = %zd\n", sizeof(cxx::ParameterDeclarationAST));
printf("sizeof(AccessDeclarationAST) = %zd\n", sizeof(cxx::AccessDeclarationAST));
printf("sizeof(ForRangeDeclarationAST) = %zd\n", sizeof(cxx::ForRangeDeclarationAST));
printf("sizeof(StructuredBindingDeclarationAST) = %zd\n", sizeof(cxx::StructuredBindingDeclarationAST));
printf("sizeof(AsmOperandAST) = %zd\n", sizeof(cxx::AsmOperandAST));
printf("sizeof(AsmQualifierAST) = %zd\n", sizeof(cxx::AsmQualifierAST));
printf("sizeof(AsmClobberAST) = %zd\n", sizeof(cxx::AsmClobberAST));
printf("sizeof(AsmGotoLabelAST) = %zd\n", sizeof(cxx::AsmGotoLabelAST));
// StatementAST
printf("sizeof(LabeledStatementAST) = %zd\n", sizeof(cxx::LabeledStatementAST));
printf("sizeof(CaseStatementAST) = %zd\n", sizeof(cxx::CaseStatementAST));
printf("sizeof(DefaultStatementAST) = %zd\n", sizeof(cxx::DefaultStatementAST));
printf("sizeof(ExpressionStatementAST) = %zd\n", sizeof(cxx::ExpressionStatementAST));
printf("sizeof(CompoundStatementAST) = %zd\n", sizeof(cxx::CompoundStatementAST));
printf("sizeof(IfStatementAST) = %zd\n", sizeof(cxx::IfStatementAST));
printf("sizeof(ConstevalIfStatementAST) = %zd\n", sizeof(cxx::ConstevalIfStatementAST));
printf("sizeof(SwitchStatementAST) = %zd\n", sizeof(cxx::SwitchStatementAST));
printf("sizeof(WhileStatementAST) = %zd\n", sizeof(cxx::WhileStatementAST));
printf("sizeof(DoStatementAST) = %zd\n", sizeof(cxx::DoStatementAST));
printf("sizeof(ForRangeStatementAST) = %zd\n", sizeof(cxx::ForRangeStatementAST));
printf("sizeof(ForStatementAST) = %zd\n", sizeof(cxx::ForStatementAST));
printf("sizeof(BreakStatementAST) = %zd\n", sizeof(cxx::BreakStatementAST));
printf("sizeof(ContinueStatementAST) = %zd\n", sizeof(cxx::ContinueStatementAST));
printf("sizeof(ReturnStatementAST) = %zd\n", sizeof(cxx::ReturnStatementAST));
printf("sizeof(CoroutineReturnStatementAST) = %zd\n", sizeof(cxx::CoroutineReturnStatementAST));
printf("sizeof(GotoStatementAST) = %zd\n", sizeof(cxx::GotoStatementAST));
printf("sizeof(DeclarationStatementAST) = %zd\n", sizeof(cxx::DeclarationStatementAST));
printf("sizeof(TryBlockStatementAST) = %zd\n", sizeof(cxx::TryBlockStatementAST));
// ExpressionAST
printf("sizeof(GeneratedLiteralExpressionAST) = %zd\n", sizeof(cxx::GeneratedLiteralExpressionAST));
printf("sizeof(CharLiteralExpressionAST) = %zd\n", sizeof(cxx::CharLiteralExpressionAST));
printf("sizeof(BoolLiteralExpressionAST) = %zd\n", sizeof(cxx::BoolLiteralExpressionAST));
printf("sizeof(IntLiteralExpressionAST) = %zd\n", sizeof(cxx::IntLiteralExpressionAST));
printf("sizeof(FloatLiteralExpressionAST) = %zd\n", sizeof(cxx::FloatLiteralExpressionAST));
printf("sizeof(NullptrLiteralExpressionAST) = %zd\n", sizeof(cxx::NullptrLiteralExpressionAST));
printf("sizeof(StringLiteralExpressionAST) = %zd\n", sizeof(cxx::StringLiteralExpressionAST));
printf("sizeof(UserDefinedStringLiteralExpressionAST) = %zd\n", sizeof(cxx::UserDefinedStringLiteralExpressionAST));
printf("sizeof(ThisExpressionAST) = %zd\n", sizeof(cxx::ThisExpressionAST));
printf("sizeof(NestedStatementExpressionAST) = %zd\n", sizeof(cxx::NestedStatementExpressionAST));
printf("sizeof(NestedExpressionAST) = %zd\n", sizeof(cxx::NestedExpressionAST));
printf("sizeof(IdExpressionAST) = %zd\n", sizeof(cxx::IdExpressionAST));
printf("sizeof(LambdaExpressionAST) = %zd\n", sizeof(cxx::LambdaExpressionAST));
printf("sizeof(FoldExpressionAST) = %zd\n", sizeof(cxx::FoldExpressionAST));
printf("sizeof(RightFoldExpressionAST) = %zd\n", sizeof(cxx::RightFoldExpressionAST));
printf("sizeof(LeftFoldExpressionAST) = %zd\n", sizeof(cxx::LeftFoldExpressionAST));
printf("sizeof(RequiresExpressionAST) = %zd\n", sizeof(cxx::RequiresExpressionAST));
printf("sizeof(VaArgExpressionAST) = %zd\n", sizeof(cxx::VaArgExpressionAST));
printf("sizeof(SubscriptExpressionAST) = %zd\n", sizeof(cxx::SubscriptExpressionAST));
printf("sizeof(CallExpressionAST) = %zd\n", sizeof(cxx::CallExpressionAST));
printf("sizeof(TypeConstructionAST) = %zd\n", sizeof(cxx::TypeConstructionAST));
printf("sizeof(BracedTypeConstructionAST) = %zd\n", sizeof(cxx::BracedTypeConstructionAST));
printf("sizeof(SpliceMemberExpressionAST) = %zd\n", sizeof(cxx::SpliceMemberExpressionAST));
printf("sizeof(MemberExpressionAST) = %zd\n", sizeof(cxx::MemberExpressionAST));
printf("sizeof(PostIncrExpressionAST) = %zd\n", sizeof(cxx::PostIncrExpressionAST));
printf("sizeof(CppCastExpressionAST) = %zd\n", sizeof(cxx::CppCastExpressionAST));
printf("sizeof(BuiltinBitCastExpressionAST) = %zd\n", sizeof(cxx::BuiltinBitCastExpressionAST));
printf("sizeof(BuiltinOffsetofExpressionAST) = %zd\n", sizeof(cxx::BuiltinOffsetofExpressionAST));
printf("sizeof(TypeidExpressionAST) = %zd\n", sizeof(cxx::TypeidExpressionAST));
printf("sizeof(TypeidOfTypeExpressionAST) = %zd\n", sizeof(cxx::TypeidOfTypeExpressionAST));
printf("sizeof(SpliceExpressionAST) = %zd\n", sizeof(cxx::SpliceExpressionAST));
printf("sizeof(GlobalScopeReflectExpressionAST) = %zd\n", sizeof(cxx::GlobalScopeReflectExpressionAST));
printf("sizeof(NamespaceReflectExpressionAST) = %zd\n", sizeof(cxx::NamespaceReflectExpressionAST));
printf("sizeof(TypeIdReflectExpressionAST) = %zd\n", sizeof(cxx::TypeIdReflectExpressionAST));
printf("sizeof(ReflectExpressionAST) = %zd\n", sizeof(cxx::ReflectExpressionAST));
printf("sizeof(UnaryExpressionAST) = %zd\n", sizeof(cxx::UnaryExpressionAST));
printf("sizeof(AwaitExpressionAST) = %zd\n", sizeof(cxx::AwaitExpressionAST));
printf("sizeof(SizeofExpressionAST) = %zd\n", sizeof(cxx::SizeofExpressionAST));
printf("sizeof(SizeofTypeExpressionAST) = %zd\n", sizeof(cxx::SizeofTypeExpressionAST));
printf("sizeof(SizeofPackExpressionAST) = %zd\n", sizeof(cxx::SizeofPackExpressionAST));
printf("sizeof(AlignofTypeExpressionAST) = %zd\n", sizeof(cxx::AlignofTypeExpressionAST));
printf("sizeof(AlignofExpressionAST) = %zd\n", sizeof(cxx::AlignofExpressionAST));
printf("sizeof(NoexceptExpressionAST) = %zd\n", sizeof(cxx::NoexceptExpressionAST));
printf("sizeof(NewExpressionAST) = %zd\n", sizeof(cxx::NewExpressionAST));
printf("sizeof(DeleteExpressionAST) = %zd\n", sizeof(cxx::DeleteExpressionAST));
printf("sizeof(CastExpressionAST) = %zd\n", sizeof(cxx::CastExpressionAST));
printf("sizeof(ImplicitCastExpressionAST) = %zd\n", sizeof(cxx::ImplicitCastExpressionAST));
printf("sizeof(BinaryExpressionAST) = %zd\n", sizeof(cxx::BinaryExpressionAST));
printf("sizeof(ConditionalExpressionAST) = %zd\n", sizeof(cxx::ConditionalExpressionAST));
printf("sizeof(YieldExpressionAST) = %zd\n", sizeof(cxx::YieldExpressionAST));
printf("sizeof(ThrowExpressionAST) = %zd\n", sizeof(cxx::ThrowExpressionAST));
printf("sizeof(AssignmentExpressionAST) = %zd\n", sizeof(cxx::AssignmentExpressionAST));
printf("sizeof(PackExpansionExpressionAST) = %zd\n", sizeof(cxx::PackExpansionExpressionAST));
printf("sizeof(DesignatedInitializerClauseAST) = %zd\n", sizeof(cxx::DesignatedInitializerClauseAST));
printf("sizeof(TypeTraitExpressionAST) = %zd\n", sizeof(cxx::TypeTraitExpressionAST));
printf("sizeof(ConditionExpressionAST) = %zd\n", sizeof(cxx::ConditionExpressionAST));
printf("sizeof(EqualInitializerAST) = %zd\n", sizeof(cxx::EqualInitializerAST));
printf("sizeof(BracedInitListAST) = %zd\n", sizeof(cxx::BracedInitListAST));
printf("sizeof(ParenInitializerAST) = %zd\n", sizeof(cxx::ParenInitializerAST));
// AST
printf("sizeof(SplicerAST) = %zd\n", sizeof(cxx::SplicerAST));
printf("sizeof(GlobalModuleFragmentAST) = %zd\n", sizeof(cxx::GlobalModuleFragmentAST));
printf("sizeof(PrivateModuleFragmentAST) = %zd\n", sizeof(cxx::PrivateModuleFragmentAST));
printf("sizeof(ModuleDeclarationAST) = %zd\n", sizeof(cxx::ModuleDeclarationAST));
printf("sizeof(ModuleNameAST) = %zd\n", sizeof(cxx::ModuleNameAST));
printf("sizeof(ModuleQualifierAST) = %zd\n", sizeof(cxx::ModuleQualifierAST));
printf("sizeof(ModulePartitionAST) = %zd\n", sizeof(cxx::ModulePartitionAST));
printf("sizeof(ImportNameAST) = %zd\n", sizeof(cxx::ImportNameAST));
printf("sizeof(InitDeclaratorAST) = %zd\n", sizeof(cxx::InitDeclaratorAST));
printf("sizeof(DeclaratorAST) = %zd\n", sizeof(cxx::DeclaratorAST));
printf("sizeof(UsingDeclaratorAST) = %zd\n", sizeof(cxx::UsingDeclaratorAST));
printf("sizeof(EnumeratorAST) = %zd\n", sizeof(cxx::EnumeratorAST));
printf("sizeof(TypeIdAST) = %zd\n", sizeof(cxx::TypeIdAST));
printf("sizeof(HandlerAST) = %zd\n", sizeof(cxx::HandlerAST));
printf("sizeof(BaseSpecifierAST) = %zd\n", sizeof(cxx::BaseSpecifierAST));
printf("sizeof(RequiresClauseAST) = %zd\n", sizeof(cxx::RequiresClauseAST));
printf("sizeof(ParameterDeclarationClauseAST) = %zd\n", sizeof(cxx::ParameterDeclarationClauseAST));
printf("sizeof(TrailingReturnTypeAST) = %zd\n", sizeof(cxx::TrailingReturnTypeAST));
printf("sizeof(LambdaSpecifierAST) = %zd\n", sizeof(cxx::LambdaSpecifierAST));
printf("sizeof(TypeConstraintAST) = %zd\n", sizeof(cxx::TypeConstraintAST));
printf("sizeof(AttributeArgumentClauseAST) = %zd\n", sizeof(cxx::AttributeArgumentClauseAST));
printf("sizeof(AttributeAST) = %zd\n", sizeof(cxx::AttributeAST));
printf("sizeof(AttributeUsingPrefixAST) = %zd\n", sizeof(cxx::AttributeUsingPrefixAST));
printf("sizeof(NewPlacementAST) = %zd\n", sizeof(cxx::NewPlacementAST));
printf("sizeof(NestedNamespaceSpecifierAST) = %zd\n", sizeof(cxx::NestedNamespaceSpecifierAST));
// TemplateParameterAST
printf("sizeof(TemplateTypeParameterAST) = %zd\n", sizeof(cxx::TemplateTypeParameterAST));
printf("sizeof(NonTypeTemplateParameterAST) = %zd\n", sizeof(cxx::NonTypeTemplateParameterAST));
printf("sizeof(TypenameTypeParameterAST) = %zd\n", sizeof(cxx::TypenameTypeParameterAST));
printf("sizeof(ConstraintTypeParameterAST) = %zd\n", sizeof(cxx::ConstraintTypeParameterAST));
// SpecifierAST
printf("sizeof(GeneratedTypeSpecifierAST) = %zd\n", sizeof(cxx::GeneratedTypeSpecifierAST));
printf("sizeof(TypedefSpecifierAST) = %zd\n", sizeof(cxx::TypedefSpecifierAST));
printf("sizeof(FriendSpecifierAST) = %zd\n", sizeof(cxx::FriendSpecifierAST));
printf("sizeof(ConstevalSpecifierAST) = %zd\n", sizeof(cxx::ConstevalSpecifierAST));
printf("sizeof(ConstinitSpecifierAST) = %zd\n", sizeof(cxx::ConstinitSpecifierAST));
printf("sizeof(ConstexprSpecifierAST) = %zd\n", sizeof(cxx::ConstexprSpecifierAST));
printf("sizeof(InlineSpecifierAST) = %zd\n", sizeof(cxx::InlineSpecifierAST));
printf("sizeof(StaticSpecifierAST) = %zd\n", sizeof(cxx::StaticSpecifierAST));
printf("sizeof(ExternSpecifierAST) = %zd\n", sizeof(cxx::ExternSpecifierAST));
printf("sizeof(ThreadLocalSpecifierAST) = %zd\n", sizeof(cxx::ThreadLocalSpecifierAST));
printf("sizeof(ThreadSpecifierAST) = %zd\n", sizeof(cxx::ThreadSpecifierAST));
printf("sizeof(MutableSpecifierAST) = %zd\n", sizeof(cxx::MutableSpecifierAST));
printf("sizeof(VirtualSpecifierAST) = %zd\n", sizeof(cxx::VirtualSpecifierAST));
printf("sizeof(ExplicitSpecifierAST) = %zd\n", sizeof(cxx::ExplicitSpecifierAST));
printf("sizeof(AutoTypeSpecifierAST) = %zd\n", sizeof(cxx::AutoTypeSpecifierAST));
printf("sizeof(VoidTypeSpecifierAST) = %zd\n", sizeof(cxx::VoidTypeSpecifierAST));
printf("sizeof(SizeTypeSpecifierAST) = %zd\n", sizeof(cxx::SizeTypeSpecifierAST));
printf("sizeof(SignTypeSpecifierAST) = %zd\n", sizeof(cxx::SignTypeSpecifierAST));
printf("sizeof(VaListTypeSpecifierAST) = %zd\n", sizeof(cxx::VaListTypeSpecifierAST));
printf("sizeof(IntegralTypeSpecifierAST) = %zd\n", sizeof(cxx::IntegralTypeSpecifierAST));
printf("sizeof(FloatingPointTypeSpecifierAST) = %zd\n", sizeof(cxx::FloatingPointTypeSpecifierAST));
printf("sizeof(ComplexTypeSpecifierAST) = %zd\n", sizeof(cxx::ComplexTypeSpecifierAST));
printf("sizeof(NamedTypeSpecifierAST) = %zd\n", sizeof(cxx::NamedTypeSpecifierAST));
printf("sizeof(AtomicTypeSpecifierAST) = %zd\n", sizeof(cxx::AtomicTypeSpecifierAST));
printf("sizeof(UnderlyingTypeSpecifierAST) = %zd\n", sizeof(cxx::UnderlyingTypeSpecifierAST));
printf("sizeof(ElaboratedTypeSpecifierAST) = %zd\n", sizeof(cxx::ElaboratedTypeSpecifierAST));
printf("sizeof(DecltypeAutoSpecifierAST) = %zd\n", sizeof(cxx::DecltypeAutoSpecifierAST));
printf("sizeof(DecltypeSpecifierAST) = %zd\n", sizeof(cxx::DecltypeSpecifierAST));
printf("sizeof(PlaceholderTypeSpecifierAST) = %zd\n", sizeof(cxx::PlaceholderTypeSpecifierAST));
printf("sizeof(ConstQualifierAST) = %zd\n", sizeof(cxx::ConstQualifierAST));
printf("sizeof(VolatileQualifierAST) = %zd\n", sizeof(cxx::VolatileQualifierAST));
printf("sizeof(RestrictQualifierAST) = %zd\n", sizeof(cxx::RestrictQualifierAST));
printf("sizeof(EnumSpecifierAST) = %zd\n", sizeof(cxx::EnumSpecifierAST));
printf("sizeof(ClassSpecifierAST) = %zd\n", sizeof(cxx::ClassSpecifierAST));
printf("sizeof(TypenameSpecifierAST) = %zd\n", sizeof(cxx::TypenameSpecifierAST));
printf("sizeof(SplicerTypeSpecifierAST) = %zd\n", sizeof(cxx::SplicerTypeSpecifierAST));
// PtrOperatorAST
printf("sizeof(PointerOperatorAST) = %zd\n", sizeof(cxx::PointerOperatorAST));
printf("sizeof(ReferenceOperatorAST) = %zd\n", sizeof(cxx::ReferenceOperatorAST));
printf("sizeof(PtrToMemberOperatorAST) = %zd\n", sizeof(cxx::PtrToMemberOperatorAST));
// CoreDeclaratorAST
printf("sizeof(BitfieldDeclaratorAST) = %zd\n", sizeof(cxx::BitfieldDeclaratorAST));
printf("sizeof(ParameterPackAST) = %zd\n", sizeof(cxx::ParameterPackAST));
printf("sizeof(IdDeclaratorAST) = %zd\n", sizeof(cxx::IdDeclaratorAST));
printf("sizeof(NestedDeclaratorAST) = %zd\n", sizeof(cxx::NestedDeclaratorAST));
// DeclaratorChunkAST
printf("sizeof(FunctionDeclaratorChunkAST) = %zd\n", sizeof(cxx::FunctionDeclaratorChunkAST));
printf("sizeof(ArrayDeclaratorChunkAST) = %zd\n", sizeof(cxx::ArrayDeclaratorChunkAST));
// UnqualifiedIdAST
printf("sizeof(NameIdAST) = %zd\n", sizeof(cxx::NameIdAST));
printf("sizeof(DestructorIdAST) = %zd\n", sizeof(cxx::DestructorIdAST));
printf("sizeof(DecltypeIdAST) = %zd\n", sizeof(cxx::DecltypeIdAST));
printf("sizeof(OperatorFunctionIdAST) = %zd\n", sizeof(cxx::OperatorFunctionIdAST));
printf("sizeof(LiteralOperatorIdAST) = %zd\n", sizeof(cxx::LiteralOperatorIdAST));
printf("sizeof(ConversionFunctionIdAST) = %zd\n", sizeof(cxx::ConversionFunctionIdAST));
printf("sizeof(SimpleTemplateIdAST) = %zd\n", sizeof(cxx::SimpleTemplateIdAST));
printf("sizeof(LiteralOperatorTemplateIdAST) = %zd\n", sizeof(cxx::LiteralOperatorTemplateIdAST));
printf("sizeof(OperatorFunctionTemplateIdAST) = %zd\n", sizeof(cxx::OperatorFunctionTemplateIdAST));
// NestedNameSpecifierAST
printf("sizeof(GlobalNestedNameSpecifierAST) = %zd\n", sizeof(cxx::GlobalNestedNameSpecifierAST));
printf("sizeof(SimpleNestedNameSpecifierAST) = %zd\n", sizeof(cxx::SimpleNestedNameSpecifierAST));
printf("sizeof(DecltypeNestedNameSpecifierAST) = %zd\n", sizeof(cxx::DecltypeNestedNameSpecifierAST));
printf("sizeof(TemplateNestedNameSpecifierAST) = %zd\n", sizeof(cxx::TemplateNestedNameSpecifierAST));
// FunctionBodyAST
printf("sizeof(DefaultFunctionBodyAST) = %zd\n", sizeof(cxx::DefaultFunctionBodyAST));
printf("sizeof(CompoundStatementFunctionBodyAST) = %zd\n", sizeof(cxx::CompoundStatementFunctionBodyAST));
printf("sizeof(TryStatementFunctionBodyAST) = %zd\n", sizeof(cxx::TryStatementFunctionBodyAST));
printf("sizeof(DeleteFunctionBodyAST) = %zd\n", sizeof(cxx::DeleteFunctionBodyAST));
// TemplateArgumentAST
printf("sizeof(TypeTemplateArgumentAST) = %zd\n", sizeof(cxx::TypeTemplateArgumentAST));
printf("sizeof(ExpressionTemplateArgumentAST) = %zd\n", sizeof(cxx::ExpressionTemplateArgumentAST));
// ExceptionSpecifierAST
printf("sizeof(ThrowExceptionSpecifierAST) = %zd\n", sizeof(cxx::ThrowExceptionSpecifierAST));
printf("sizeof(NoexceptSpecifierAST) = %zd\n", sizeof(cxx::NoexceptSpecifierAST));
// RequirementAST
printf("sizeof(SimpleRequirementAST) = %zd\n", sizeof(cxx::SimpleRequirementAST));
printf("sizeof(CompoundRequirementAST) = %zd\n", sizeof(cxx::CompoundRequirementAST));
printf("sizeof(TypeRequirementAST) = %zd\n", sizeof(cxx::TypeRequirementAST));
printf("sizeof(NestedRequirementAST) = %zd\n", sizeof(cxx::NestedRequirementAST));
// NewInitializerAST
printf("sizeof(NewParenInitializerAST) = %zd\n", sizeof(cxx::NewParenInitializerAST));
printf("sizeof(NewBracedInitializerAST) = %zd\n", sizeof(cxx::NewBracedInitializerAST));
// MemInitializerAST
printf("sizeof(ParenMemInitializerAST) = %zd\n", sizeof(cxx::ParenMemInitializerAST));
printf("sizeof(BracedMemInitializerAST) = %zd\n", sizeof(cxx::BracedMemInitializerAST));
// LambdaCaptureAST
printf("sizeof(ThisLambdaCaptureAST) = %zd\n", sizeof(cxx::ThisLambdaCaptureAST));
printf("sizeof(DerefThisLambdaCaptureAST) = %zd\n", sizeof(cxx::DerefThisLambdaCaptureAST));
printf("sizeof(SimpleLambdaCaptureAST) = %zd\n", sizeof(cxx::SimpleLambdaCaptureAST));
printf("sizeof(RefLambdaCaptureAST) = %zd\n", sizeof(cxx::RefLambdaCaptureAST));
printf("sizeof(RefInitLambdaCaptureAST) = %zd\n", sizeof(cxx::RefInitLambdaCaptureAST));
printf("sizeof(InitLambdaCaptureAST) = %zd\n", sizeof(cxx::InitLambdaCaptureAST));
// ExceptionDeclarationAST
printf("sizeof(EllipsisExceptionDeclarationAST) = %zd\n", sizeof(cxx::EllipsisExceptionDeclarationAST));
printf("sizeof(TypeExceptionDeclarationAST) = %zd\n", sizeof(cxx::TypeExceptionDeclarationAST));
// AttributeSpecifierAST
printf("sizeof(CxxAttributeAST) = %zd\n", sizeof(cxx::CxxAttributeAST));
printf("sizeof(GccAttributeAST) = %zd\n", sizeof(cxx::GccAttributeAST));
printf("sizeof(AlignasAttributeAST) = %zd\n", sizeof(cxx::AlignasAttributeAST));
printf("sizeof(AlignasTypeAttributeAST) = %zd\n", sizeof(cxx::AlignasTypeAttributeAST));
printf("sizeof(AsmAttributeAST) = %zd\n", sizeof(cxx::AsmAttributeAST));
// AttributeTokenAST
printf("sizeof(ScopedAttributeTokenAST) = %zd\n", sizeof(cxx::ScopedAttributeTokenAST));
printf("sizeof(SimpleAttributeTokenAST) = %zd\n", sizeof(cxx::SimpleAttributeTokenAST));
printf("sizeof(SourceLocation) = %zd\n", sizeof(cxx::SourceLocation));
printf("sizeof(AST) = %zd\n", sizeof(cxx::AST));
printf("sizeof(MyAST) = %zd\n", sizeof(cxx::MyAST));
printf("sizeof(ExpressionAST) = %zd\n", sizeof(cxx::ExpressionAST));
printf("sizeof(MyExpressionAST) = %zd\n", sizeof(cxx::MyExpressionAST));
printf("sizeof(MyMemberExpressionAST) = %zd\n", sizeof(cxx::MyMemberExpressionAST));
printf("sizeof(MyMemberExpressionAST2) = %zd\n", sizeof(cxx::MyMemberExpressionAST2));
printf("sizeof(MyTemplateDeclarationAST) = %zd\n", sizeof(cxx::MyTemplateDeclarationAST));
printf("sizeof(MyVaListTypeSpecifierAST) = %zd\n", sizeof(cxx::MyVaListTypeSpecifierAST));
return 0;
}
/*
sizeof(SourceLocation) = 4
sizeof(AST) = 16
sizeof(MyAST) = 16
sizeof(ExpressionAST) = 24
sizeof(MyExpressionAST) = 24
sizeof(MyMemberExpressionAST) = 72
sizeof(MyMemberExpressionAST2) = 72
sizeof(MyTemplateDeclarationAST) = 56
sizeof(MyVaListTypeSpecifierAST) = 24
sizeof(LambdaExpressionAST) = 168
sizeof(TemplateTypeParameterAST) = 104 -4
sizeof(AsmDeclarationAST) = 96
sizeof(ClassSpecifierAST) = 96 -12
sizeof(EnumSpecifierAST) = 96 -8
sizeof(NamespaceDefinitionAST) = 96 -12
sizeof(NewExpressionAST) = 88 -8
sizeof(ParameterDeclarationAST) = 88 -8
sizeof(AliasDeclarationAST) = 80 -8
sizeof(ForRangeStatementAST) = 80 -8
sizeof(ForStatementAST) = 80 -8
sizeof(FunctionDeclaratorChunkAST) = 80 -12
sizeof(IfStatementAST) = 80 -8
sizeof(MemberExpressionAST) = 80
sizeof(BaseSpecifierAST) = 72 -8
sizeof(BuiltinOffsetofExpressionAST) = 72 -8
sizeof(ConstraintTypeParameterAST) = 72
sizeof(DeductionGuideAST) = 72
sizeof(FoldExpressionAST) = 72 -8
sizeof(OpaqueEnumDeclarationAST) = 72 -8
sizeof(StructuredBindingDeclarationAST) = 72 -8
sizeof(TypenameTypeParameterAST) = 72
sizeof(AsmOperandAST) = 64 -8
sizeof(BuiltinBitCastExpressionAST) = 64 -8
sizeof(ConditionalExpressionAST) = 64 -8
sizeof(ConditionExpressionAST) = 64
sizeof(CppCastExpressionAST) = 64
sizeof(ElaboratedTypeSpecifierAST) = 64 -8
sizeof(FunctionDefinitionAST) = 64
sizeof(GeneratedLiteralExpressionAST) = 64 ?8
sizeof(IdExpressionAST) = 64 -4
sizeof(RequiresExpressionAST) = 64
sizeof(SpliceMemberExpressionAST) = 64
sizeof(SwitchStatementAST) = 64 -4
sizeof(TemplateDeclarationAST) = 64
sizeof(VaArgExpressionAST) = 64
sizeof(AssignmentExpressionAST) = 56
sizeof(BinaryExpressionAST) = 56
sizeof(CallExpressionAST) = 56
sizeof(CastExpressionAST) = 56
sizeof(CompoundRequirementAST) = 56
sizeof(ConceptDefinitionAST) = 56
sizeof(ConstevalIfStatementAST) = 56
sizeof(EnumeratorAST) = 56
sizeof(IdDeclaratorAST) = 56
sizeof(LeftFoldExpressionAST) = 56
sizeof(ModuleDeclarationAST) = 56
sizeof(NamedTypeSpecifierAST) = 56
sizeof(NamespaceAliasDefinitionAST) = 56
sizeof(ParenMemInitializerAST) = 56
sizeof(RightFoldExpressionAST) = 56
sizeof(SimpleDeclarationAST) = 56
sizeof(SimpleNestedNameSpecifierAST) = 56
sizeof(SimpleTemplateIdAST) = 56
sizeof(SizeofPackExpressionAST) = 56
sizeof(StaticAssertDeclarationAST) = 56
sizeof(SubscriptExpressionAST) = 56
sizeof(TemplateNestedNameSpecifierAST) = 56
sizeof(TypeConstraintAST) = 56
sizeof(TypeConstructionAST) = 56
sizeof(UsingDeclaratorAST) = 56
sizeof(UsingDirectiveAST) = 56
sizeof(WhileStatementAST) = 56
sizeof(AlignasAttributeAST) = 48
sizeof(AlignasTypeAttributeAST) = 48
sizeof(AlignofTypeExpressionAST) = 48
sizeof(BracedInitListAST) = 48
sizeof(BracedMemInitializerAST) = 48
sizeof(CxxAttributeAST) = 48
sizeof(DecltypeSpecifierAST) = 48
sizeof(DeleteExpressionAST) = 48
sizeof(DesignatedInitializerClauseAST) = 48
sizeof(DoStatementAST) = 48
sizeof(HandlerAST) = 48
sizeof(InitDeclaratorAST) = 48
sizeof(LinkageSpecificationAST) = 48
sizeof(LiteralOperatorTemplateIdAST) = 48
sizeof(ModuleUnitAST) = 48
sizeof(NamespaceReflectExpressionAST) = 48
sizeof(NestedExpressionAST) = 48
sizeof(NestedStatementExpressionAST) = 48
sizeof(NoexceptExpressionAST) = 48
sizeof(OperatorFunctionTemplateIdAST) = 48
sizeof(ParameterDeclarationClauseAST) = 48
sizeof(ParenInitializerAST) = 48
sizeof(PtrToMemberOperatorAST) = 48
sizeof(SizeofTypeExpressionAST) = 48
sizeof(TryStatementFunctionBodyAST) = 48
sizeof(TypeidExpressionAST) = 48
sizeof(TypeidOfTypeExpressionAST) = 48
sizeof(TypenameSpecifierAST) = 48
sizeof(TypeRequirementAST) = 48
sizeof(TypeTraitExpressionAST) = 48
sizeof(UnaryExpressionAST) = 48
sizeof(AlignofExpressionAST) = 40
sizeof(ArrayDeclaratorChunkAST) = 40
sizeof(AsmAttributeAST) = 40
sizeof(AtomicTypeSpecifierAST) = 40
sizeof(AttributeAST) = 40
sizeof(AwaitExpressionAST) = 40
sizeof(BitfieldDeclaratorAST) = 40
sizeof(BracedTypeConstructionAST) = 40
sizeof(CharLiteralExpressionAST) = 40
sizeof(CompoundStatementAST) = 40
sizeof(DeclaratorAST) = 40
sizeof(DecltypeNestedNameSpecifierAST) = 40
sizeof(EqualInitializerAST) = 40
sizeof(ExplicitSpecifierAST) = 40
sizeof(ExportCompoundDeclarationAST) = 40
sizeof(FloatLiteralExpressionAST) = 40
sizeof(ImplicitCastExpressionAST) = 40
sizeof(InitLambdaCaptureAST) = 40
sizeof(IntLiteralExpressionAST) = 40
sizeof(LiteralOperatorIdAST) = 40
sizeof(ModuleImportDeclarationAST) = 40
sizeof(ModuleNameAST) = 40
sizeof(ModuleQualifierAST) = 40
sizeof(NestedNamespaceSpecifierAST) = 40
sizeof(NoexceptSpecifierAST) = 40
sizeof(NonTypeTemplateParameterAST) = 40
sizeof(PackExpansionExpressionAST) = 40
sizeof(PostIncrExpressionAST) = 40
sizeof(PrivateModuleFragmentAST) = 40
sizeof(RefInitLambdaCaptureAST) = 40
sizeof(ReflectExpressionAST) = 40
sizeof(ScopedAttributeTokenAST) = 40
sizeof(SizeofExpressionAST) = 40
sizeof(SplicerAST) = 40
sizeof(StringLiteralExpressionAST) = 40
sizeof(ThrowExpressionAST) = 40
sizeof(TypeExceptionDeclarationAST) = 40
sizeof(TypeIdAST) = 40
sizeof(TypeIdReflectExpressionAST) = 40
sizeof(UnderlyingTypeSpecifierAST) = 40
sizeof(UserDefinedStringLiteralExpressionAST) = 40
sizeof(YieldExpressionAST) = 40
sizeof(AttributeDeclarationAST) = 32
sizeof(BoolLiteralExpressionAST) = 32
sizeof(CaseStatementAST) = 32
sizeof(CompoundStatementFunctionBodyAST) = 32
sizeof(CoroutineReturnStatementAST) = 32
sizeof(DecltypeAutoSpecifierAST) = 32
sizeof(ExplicitInstantiationAST) = 32
sizeof(ExpressionStatementAST) = 32
sizeof(GccAttributeAST) = 32
sizeof(GlobalModuleFragmentAST) = 32
sizeof(GlobalNestedNameSpecifierAST) = 32
sizeof(GlobalScopeReflectExpressionAST) = 32
sizeof(GotoStatementAST) = 32
sizeof(ImportNameAST) = 32
sizeof(LabeledStatementAST) = 32
sizeof(NestedDeclaratorAST) = 32
sizeof(NestedRequirementAST) = 32
sizeof(NewParenInitializerAST) = 32
sizeof(NewPlacementAST) = 32
sizeof(NullptrLiteralExpressionAST) = 32
sizeof(OperatorFunctionIdAST) = 32
sizeof(PlaceholderTypeSpecifierAST) = 32
sizeof(PointerOperatorAST) = 32
sizeof(ReferenceOperatorAST) = 32
sizeof(RefLambdaCaptureAST) = 32
sizeof(ReturnStatementAST) = 32
sizeof(SimpleLambdaCaptureAST) = 32
sizeof(SimpleRequirementAST) = 32
sizeof(SpliceExpressionAST) = 32
sizeof(ThisExpressionAST) = 32
sizeof(TryBlockStatementAST) = 32
sizeof(UsingDeclarationAST) = 32
sizeof(UsingEnumDeclarationAST) = 32
sizeof(AccessDeclarationAST) = 24
sizeof(AsmClobberAST) = 24
sizeof(AsmGotoLabelAST) = 24
sizeof(AsmQualifierAST) = 24
sizeof(AttributeArgumentClauseAST) = 24
sizeof(AttributeUsingPrefixAST) = 24
sizeof(BreakStatementAST) = 24
sizeof(ContinueStatementAST) = 24
sizeof(ConversionFunctionIdAST) = 24
sizeof(DeclarationStatementAST) = 24
sizeof(DecltypeIdAST) = 24
sizeof(DefaultFunctionBodyAST) = 24
sizeof(DefaultStatementAST) = 24
sizeof(DeleteFunctionBodyAST) = 24
sizeof(DerefThisLambdaCaptureAST) = 24
sizeof(DestructorIdAST) = 24
sizeof(ExportDeclarationAST) = 24
sizeof(ExpressionTemplateArgumentAST) = 24
sizeof(FloatingPointTypeSpecifierAST) = 24
sizeof(GeneratedTypeSpecifierAST) = 24
sizeof(IntegralTypeSpecifierAST) = 24
sizeof(LambdaSpecifierAST) = 24
sizeof(ModulePartitionAST) = 24
sizeof(NameIdAST) = 24
sizeof(NewBracedInitializerAST) = 24
sizeof(ParameterPackAST) = 24
sizeof(RequiresClauseAST) = 24
sizeof(SignTypeSpecifierAST) = 24
sizeof(SimpleAttributeTokenAST) = 24
sizeof(SizeTypeSpecifierAST) = 24
sizeof(SplicerTypeSpecifierAST) = 24
sizeof(ThrowExceptionSpecifierAST) = 24
sizeof(TrailingReturnTypeAST) = 24
sizeof(TranslationUnitAST) = 24
sizeof(TypeTemplateArgumentAST) = 24
sizeof(VaListTypeSpecifierAST) = 24
sizeof(AutoTypeSpecifierAST) = 16
sizeof(ComplexTypeSpecifierAST) = 16
sizeof(ConstevalSpecifierAST) = 16
sizeof(ConstexprSpecifierAST) = 16
sizeof(ConstinitSpecifierAST) = 16
sizeof(ConstQualifierAST) = 16
sizeof(EllipsisExceptionDeclarationAST) = 16
sizeof(EmptyDeclarationAST) = 16
sizeof(ExternSpecifierAST) = 16
sizeof(ForRangeDeclarationAST) = 16
sizeof(FriendSpecifierAST) = 16
sizeof(InlineSpecifierAST) = 16
sizeof(MutableSpecifierAST) = 16
sizeof(RestrictQualifierAST) = 16
sizeof(StaticSpecifierAST) = 16
sizeof(ThisLambdaCaptureAST) = 16
sizeof(ThreadLocalSpecifierAST) = 16
sizeof(ThreadSpecifierAST) = 16
sizeof(TypedefSpecifierAST) = 16
sizeof(VirtualSpecifierAST) = 16
sizeof(VoidTypeSpecifierAST) = 16
sizeof(VolatileQualifierAST) = 16
*/
Obs.: Edited adding a virtual method to MyAst to be more realistics in terms of sizes.
Spliting members.forEach in function gen_ast_h:
members.forEach((m) => {
switch (m.kind) {
case "token":
emit(` SourceLocation ${m.name};`);
break;
}
});
members.forEach((m) => {
switch (m.kind) {
case "node":
emit(` ${m.type}* ${m.name} = nullptr;`);
break;
case "node-list":
emit(` List<${m.type}*>* ${m.name} = nullptr;`);
break;
case "token-list":
emit(` List<SourceLocation>* ${m.name} = nullptr;`);
break;
case "attribute": {
let s = " ";
if (m.cv) s = `${s}${m.cv} `;
s = `${s}${m.type}${m.ptrOps} ${m.name}`;
if (m.initializer) s = `${s} = ${m.initializer}`;
s = `${s};`;
emit(s);
break;
}
}
});
Gives some gains but still have more to gain:
sizeof(SimpleDeclarationAST) = 56 -> 48
sizeof(AsmDeclarationAST) = 96 -> 88
sizeof(UsingDirectiveAST) = 56 -> 48
sizeof(AliasDeclarationAST) = 80 -> 72
sizeof(OpaqueEnumDeclarationAST) = 72 -> 64
sizeof(TemplateDeclarationAST) = 64 -> 56
sizeof(DeductionGuideAST) = 72 -> 64
sizeof(ExportCompoundDeclarationAST) = 40 -> 32
sizeof(NamespaceDefinitionAST) = 96 -> 80
sizeof(AttributeDeclarationAST) = 32 -> 24
sizeof(ParameterDeclarationAST) = 88 -> 80
sizeof(StructuredBindingDeclarationAST) = 72 -> 64
sizeof(ExpressionStatementAST) = 32 -> 24
sizeof(IfStatementAST) = 80 -> 72
sizeof(SwitchStatementAST) = 64 -> 56
sizeof(WhileStatementAST) = 56 -> 48
sizeof(ForRangeStatementAST) = 80 -> 72
sizeof(ForStatementAST) = 80 -> 72
sizeof(NestedStatementExpressionAST) = 48 -> 40
sizeof(NestedExpressionAST) = 48 -> 40
sizeof(LambdaExpressionAST) = 168 -> 160
sizeof(VaArgExpressionAST) = 64 -> 56
sizeof(SubscriptExpressionAST) = 56 -> 48
sizeof(CallExpressionAST) = 56 -> 48
sizeof(TypeConstructionAST) = 56 -> 48
sizeof(MemberExpressionAST) = 80 -> 72
sizeof(PostIncrExpressionAST) = 40 -> 48
sizeof(BuiltinBitCastExpressionAST) = 64 -> 56
sizeof(BuiltinOffsetofExpressionAST) = 72 -> 64
sizeof(NewExpressionAST) = 88 -> 80
sizeof(CastExpressionAST) = 56 -> 48
sizeof(ConditionalExpressionAST) = 64 -> 56
sizeof(TypeTraitExpressionAST) = 48 -> 56
sizeof(ParenInitializerAST) = 48 -> 40
sizeof(ModuleDeclarationAST) = 56 -> 48
sizeof(ModuleNameAST) = 40 -> 32
sizeof(HandlerAST) = 48 -> 40
sizeof(TypeConstraintAST) = 56 -> 48
sizeof(AttributeAST) = 40 -> 32
sizeof(ExplicitSpecifierAST) = 40 -> 32
sizeof(NamedTypeSpecifierAST) = 56 -> 48
sizeof(AtomicTypeSpecifierAST) = 40 -> 32
sizeof(UnderlyingTypeSpecifierAST) = 40 -> 32
sizeof(DecltypeSpecifierAST) = 48 -> 40
sizeof(EnumSpecifierAST) = 96 -> 88
sizeof(PtrToMemberOperatorAST) = 48 -> 40
sizeof(BitfieldDeclaratorAST) = 40 -> 32
sizeof(IdDeclaratorAST) = 56 -> 48
sizeof(FunctionDeclaratorChunkAST) = 80 -> 72
sizeof(SimpleTemplateIdAST) = 56 -> 48
sizeof(LiteralOperatorTemplateIdAST) = 48 -> 40
sizeof(OperatorFunctionTemplateIdAST) = 48 -> 40
sizeof(SimpleNestedNameSpecifierAST) = 56 -> 48
sizeof(NoexceptSpecifierAST) = 40 -> 32
sizeof(SimpleRequirementAST) = 32 -> 24
sizeof(CompoundRequirementAST) = 56 -> 48
sizeof(ParenMemInitializerAST) = 56 -> 48
sizeof(BracedMemInitializerAST) = 48 -> 40
After testing with the memory aligned ast.h (cxx-mal) with sqlite3.c the memory size reduction is only 10MB around 4.5% reduction:
/usr/bin/time ./cxx-mal -toolchain linuxC sqlite3-0.c
1.10user 0.11system 0:01.22elapsed 99%CPU (0avgtext+0avgdata 178552maxresident)k
0inputs+0outputs (0major+68286minor)pagefaults 0swaps
/usr/bin/time cppcxx -toolchain linuxC sqlite3-0.c
1.11user 0.10system 0:01.22elapsed 99%CPU (0avgtext+0avgdata 188340maxresident)k
0inputs+0outputs (0major+70760minor)pagefaults 0swaps
Thanks @mingodad, I'm aware of the gaps between the memory layout of the AST nodes’ However, I do rely on the order of these members. Removing the gaps will save memory, but the real savings and speedup will come from reducing the memory waste of AST nodes created during backtracking (and the different size in memory of the typed AST nodes). After I finalized the AST attributes and their subtypes, I’ll probably write a small tool to rewrite the AST using a list of slots and reuses the nodes during backtracking, I will re-add a fully typed interface on top of it. For now, it’s better to maintain the interleaved order because many parts of the parser need to know the order of these fields (e.g. when walking the tree structure, code rewriting, incremental parsing, and so on.).