feat: add support for C++20 modules (#174)

[alfaix]

This PR adds support for C++20 modules.

(Semi-)formal syntax description can be found on cppreference.

While implementing support for modules, I also noticed that class template instantiation, as well as explicit function (or class) templates instantiation declarations (i.e., extern (template_instatiation)) were not parsed correctly. I fixed that in the second commit of three. Please let me know if it would be preferrable to split this in two PRs.

I'm not a tree-sitter expert, I just want to experiment with C++20 modules in my neovim. Thus, the new rules may have a couple issues. The ones that stick out to me are:

1. The PR introduces a few LR1 conflicts. I'm not sure if they can be avoided by precedence - what I tried failed.
2. optional($.export_declaration) needed to be added to (almost) every top-level declaration. Not sure if there is a more elegant way of doing it, as many of them are not grouped together and introducing such a group now would break old queries. I wanted to keep the export specifier as part of the declaration, so introducing a new export statement felt like a worse solution.

Summary:

• introduce new keywords: module, export, import
• add support for new C++20 statements: module, export, import
• fix template_instantiation to match class instantiation, as well as instantiation declarations
• add the export keyword as an optional part of all exportable declarations and definitions

Resolves: #174

[berchn]

Has anyone had a chance to look at this MR yet?

[alfaix]

I merged the most recent master in case someone does want to take a look at the PR, but the build is failing. AFAICS, it's failing for the same reason as the master build, and I'm not sure how to fix it.

[XVilka]

@alfaix please rebase, to trigger the updated CI checks.

[alfaix]

Rebased, ran tests, seems to be fine. The commit history is a little bit messy, but assuming you'll squash shouldn't matter. Let me know if you'd like a separate PR with cleaner history.

[amaanq]

This increases state count too much, I believe this can be refactored and improved

[alfaix]

I don't know much about tree sitter so I don't really understand what increased the state count. Is it the $.export_specifier everywhere or the way module statements are parsed? Could you please give me some pointers/links to understand this better?

If it's the export thing then I guess it can be changed to a single export_statement but that will turn half the code into export statements instead of declarations, and would be annoying to work with, right?

[neg-c]

@alfaix there you go: https://github.com/tree-sitter/tree-sitter/wiki/Tips-and-Tricks-for-a-grammar-author

[ghost]

Working Draft Programming Languages — C++

(Generated on 2024-04-22 from the LaTeX sources by cxxdraft-htmlgen. This is not an ISO publication. For historical versions of the document, see Tim Song's cppwp page.)

Note: this is an early draft. It's known to be incomplet and incorrekt, and it has lots of bad formatting.

Contents 1 Scope [intro.scope] 2 Normative references [intro.refs] 3 Terms and definitions [intro.defs] 4 General principles [intro] 5 Lexical conventions [lex] 6 Basics [basic] 7 Expressions [expr] 8 Statements [stmt.stmt] 9 Declarations [dcl.dcl] 10 Modules [module] 11 Classes [class] 12 Overloading [over] 13 Templates [temp] 13.1 Preamble [temp.pre] 13.2 Template parameters [temp.param] 13.3 Names of template specializations [temp.names] 13.4 Template arguments [temp.arg] 13.4.1 General [temp.arg.general] 13.4.2 Template type arguments [temp.arg.type] 13.4.3 Template non-type arguments [temp.arg.nontype] 13.4.4 Template template arguments [temp.arg.template] 13.5 Template constraints [temp.constr] 13.5.1 General [temp.constr.general] 13.5.2 Constraints [temp.constr.constr] 13.5.2.1 General [temp.constr.constr.general] 13.5.2.2 Logical operations [temp.constr.op] 13.5.2.3 Atomic constraints [temp.constr.atomic] 13.5.3 Constrained declarations [temp.constr.decl] 13.5.4 Constraint normalization [temp.constr.normal] 13.5.5 Partial ordering by constraints [temp.constr.order] 13.6 Type equivalence [temp.type] 13.7 Template declarations [temp.decls] 13.7.1 General [temp.decls.general] 13.7.2 Class templates [temp.class] 13.7.2.1 General [temp.class.general] 13.7.2.2 Member functions of class templates [temp.mem.func] 13.7.2.3 Deduction guides [temp.deduct.guide] 13.7.2.4 Member classes of class templates [temp.mem.class] 13.7.2.5 Static data members of class templates [temp.static] 13.7.2.6 Enumeration members of class templates [temp.mem.enum] 13.7.3 Member templates [temp.mem] 13.7.4 Variadic templates [temp.variadic] 13.7.5 Friends [temp.friend] 13.7.6 Partial specialization [temp.spec.partial] 13.7.6.1 General [temp.spec.partial.general] 13.7.6.2 Matching of partial specializations [temp.spec.partial.match] 13.7.6.3 Partial ordering of partial specializations [temp.spec.partial.order] 13.7.6.4 Members of class template partial specializations [temp.spec.partial.member] 13.7.7 Function templates [temp.fct] 13.7.7.1 General [temp.fct.general] 13.7.7.2 Function template overloading [temp.over.link] 13.7.7.3 Partial ordering of function templates [temp.func.order] 13.7.8 Alias templates [temp.alias] 13.7.9 Concept definitions [temp.concept] 13.8 Name resolution [temp.res] 13.8.1 General [temp.res.general] 13.8.2 Locally declared names [temp.local] 13.8.3 Dependent names [temp.dep] 13.8.3.1 General [temp.dep.general] 13.8.3.2 Dependent types [temp.dep.type] 13.8.3.3 Type-dependent expressions [temp.dep.expr] 13.8.3.4 Value-dependent expressions [temp.dep.constexpr] 13.8.3.5 Dependent template arguments [temp.dep.temp] 13.8.4 Dependent name resolution [temp.dep.res] 13.8.4.1 Point of instantiation [temp.point] 13.8.4.2 Candidate functions [temp.dep.candidate] 13.9 Template instantiation and specialization [temp.spec] 13.9.1 General [temp.spec.general] 13.9.2 Implicit instantiation [temp.inst] 13.9.3 Explicit instantiation [temp.explicit] 13.9.4 Explicit specialization [temp.expl.spec] 13.10 Function template specializations [temp.fct.spec] 13.10.1 General [temp.fct.spec.general] 13.10.2 Explicit template argument specification [temp.arg.explicit] 13.10.3 Template argument deduction [temp.deduct] 13.10.3.1 General [temp.deduct.general] 13.10.3.2 Deducing template arguments from a function call [temp.deduct.call] 13.10.3.3 Deducing template arguments taking the address of a function template [temp.deduct.funcaddr] 13.10.3.4 Deducing conversion function template arguments [temp.deduct.conv] 13.10.3.5 Deducing template arguments during partial ordering [temp.deduct.partial] 13.10.3.6 Deducing template arguments from a type [temp.deduct.type] 13.10.3.7 Deducing template arguments from a function declaration [temp.deduct.decl] 13.10.4 Overload resolution [temp.over] 14 Exception handling [except] 15 Preprocessing directives [cpp] 16 Library introduction [library] 17 Language support library [support] 18 Concepts library [concepts] 19 Diagnostics library [diagnostics] 20 Memory management library [mem] 21 Metaprogramming library [meta] 22 General utilities library [utilities] 23 Strings library [strings] 24 Containers library [containers] 25 Iterators library [iterators] 26 Ranges library [ranges] 27 Algorithms library [algorithms] 28 Numerics library [numerics] 29 Time library [time] 30 Localization library [localization] 31 Input/output library [input.output] 32 Regular expressions library [re] 33 Concurrency support library [thread] Annex A (informative) Grammar summary [gram] Annex B (normative) Implementation quantities [implimits] Annex C (informative) Compatibility [diff] Annex D (normative) Compatibility features [depr] Annex E (informative) Conformance with UAX #31 [uaxid] Bibliography Index Index of grammar productions Index of library headers Index of library names Index of library concepts Index of implementation-defined behavior

[Tomcat-42]

Whats the current status on this?

[Tomcat-42]

@amaanq @alfaix

[alfaix]

@Tomcat-42 I just use my branch in my neovim and it works. When I have the time, I can go back to this and work on reducing the state count, but for now have other priorities.

If you have the time and the energy - feel free to take over :)

[Tomcat-42]

Thanks for your attention, @alfaix. As you suggested, I continued your work in #266.

Now, regarding the class template instantiation and explicit template instantiation declaration issues that you found, will you have time to open another PR to address them separately?

[alfaix]

@Tomcat-42 sure, will do