C++: concepts

[mattkretz]

The name of the parser: C++

The command line you used to run ctags:

$ ctags

The content of input file:

template <typename T> concept foo = true;

The tags output you are not satisfied with: It's empty.

The tags output you expect:

foo     test.h  /^template <typename T> concept foo = true;$/

The version of ctags:

$ ctags --version
Universal Ctags 6.1.0(), Copyright (C) 2015-2023 Universal Ctags Team
Universal Ctags is derived from Exuberant Ctags.
Exuberant Ctags 5.8, Copyright (C) 1996-2009 Darren Hiebert
  Compiled: Jul 18 2024, 10:25:58
  URL: https://ctags.io/
  Output version: 0.0
  Optional compiled features: +wildcards, +regex, +iconv, +option-directory, +xpath, +packcc, +optscript, +pcre2

How do you get ctags binary: Building it locally from git master.

I'm willing to give a pull request a shot. cxxParserParseBlockInternal, handle CXXKeywordCONCEPT in the switch, call a new cxxParserParseConcept function. That's how far I got. I'd start with a copy of the cxxParserParseEnum function and try to adapt that. Does that sound right? Any guidance?

[masatake]

Thank you for reporting. The original author of the C++ parser is absent these days. So I cannot give any advice quickly. So hereafter, I will write more generic advice.

1. Write your ideal tag output

foo     test.h  /^template <typename T> concept foo = true;$/

This is broken as a tags file. At least kind information is needed.

yamato@dev64:~/var/ctags-github$ PS1='$ '
$ cat input.c
int x;
$ ./ctags -o - input.c
x	input.c	/^int x;$/;"	v	typeref:typename:int
$ ./ctags -o - --fields=+K input.c
x	input.c	/^int x;$/;"	variable	typeref:typename:int
$ ./ctags -o - --fields=+Kz input.c
x	input.c	/^int x;$/;"	kind:variable	typeref:typename:int

In the above example, variable (or v) is the kind for x. You may want to define a new kind for foo. Maybe concept (or C). I can implement code to support the concept. However, I can do it only if you, a person who knows C++, give me the direction. Designing an ideal tag output for concept is needed.

In addition, like typeref:typename:int, I wonder if you may want to add more fields to the tag of foo.

/ctags --list-fields shows the available fields. If you can find fields suitable for the concept, show a tag output for foo with the fields.

This step is the most important.

2. Convert the ideal tag output to test cases of Units. See https://docs.ctags.io/en/latest/testing-parser.html
3. Exnted the C++ parser If the test cases are available, anyone can do this, even if the one doesn't know C++.

[mattkretz]

Thanks. I believe that, as a first approximation, we could tag a concept as a type (Since a concept is a generalization of a type.) But yes, tagging this correctly instead of a type should be the correct (long-term) solution.

On designing the ideal tag output... I have no idea so I'm going to guess what could be relevant.

1. A concept is always a template. There's an unconditional template-head preceding the concept keyword in the C++ grammar. Thus, one additional field to add to the concept name could be the template parameters? Examples: std::integral<T> is true if T is an integral type. std::constructible_from<T, Args...> is true if T(Args...) is a constructor call (initializes a variable). template <std::integral T> void f(T) declares a template parameter T for f which must be integral.
2. A concept can only be nested in a namespace, not in a struct/class. It also cannot be specialized. This makes it much simpler compared to types. But IIUC this is only relevant to the parser implementation, not to the output.

Thus, the most verbose output for

template <typename T> concept foo = true;

would be

foo     test.h  /^template <typename T> concept foo = true;$/;"	kind:concept	template:<typename T>

Does that make sense?

I'll look into writing a simple test.

[mattkretz]

Would the following test make sense:

input.cpp:

template <typename T>
concept addable = requires(T a, T b) { {a + b}; };

expected.tags:

addable	input.cpp	/^concept addable = requires(T a, T b) { {a + b}; };$/;"	concept	file:	template:<typename T>
T	input.cpp	/^template <typename T>$/;"	tparam	scope:concept:addable	typeref:meta:class

Is it correct, that file: is empty? I copied it from using-in-template.d.

args.ctags:

--sort=no
--kinds-C++=*
--fields=+stKZ
--fields-C++=+{template}

[masatake]

I have studied "concepts" on the page https://cpprefjp.github.io/lang/cpp20/concepts.html (Japanese).

I found supporting "concepts" is not an easy task.

Extracting the definition of a concept may be simple. However, making the parser work well with input using concepts may require many modifications.

Let's focus on parsing definitions.

I read your comments. I agree with your analysis. About file:, maybe correct. The semantics of file: is not defined well. I guess it was designed for relatively simple language like C.

using-in-template.d should be using-in-template.b. Till we add new code for concept, the test case is never passed. .b is for such a test case. With .b, we can merge a test case without breaking the test results on our CI/CD pipeline.

So, we will make two pull requests.

1. .b test case(s) based on your comment https://github.com/universal-ctags/ctags/issues/4065#issuecomment-2324979534
2. code extracting concepts and the result of git mv Units/parser-cxx.r/using-in-template.b Units/parser-cxx.r/using-in-template.d.

It seems that people call this approach test-driven development.

--sort=no
--kinds-C++=*
--fields=+stKZ
--fields-C++=+{template}

Looks good. In addition, could you add {end} (or e), {line} (or n)?

Let's add more new lines for testing end: field:

$ cat -n input.cpp
     1	template <typename T>
     2	concept addable = requires(T a, T b) {
     3	  {a + b};
     4	};

expected.tags:

addable	input.cpp	/^concept addable = requires(T a, T b) { {a + b}; };$/;"	concept	line:2	file:	end:4	template:<typename T>
T	input.cpp	/^template <typename T>$/;"	tparam	line:1	scope:concept:addable	typeref:meta:class

args.ctags:

--sort=no
--kinds-C++=*
--fields=+stKZne
--fields-C++=+{template}

In the future, it will be nice if the parser can extract a and b.

a	input.cpp	/^concept addable = requires(T a, T b) { {a + b}; };$/;"	parameter	line:2	scope:concept:addable	typeref:typename:T
b	input.cpp	/^concept addable = requires(T a, T b) { {a + b}; };$/;"	parameter	line:2	scope:concept:addable	typeref:typename:T

@mattkretz, could you make the .b pull request? Including a and b is optional.