Request: pseudo tags to record command line arguments

[AmaiKinono]

For now, most ctags client tools create tags file using a fixed command line. Users can't customize the command line for different projects.

If the command line arguments are recorded in the tags file itself, we could recreate it without knowing more information (though TAG_PROC_CWD is needed). In this way, a client tool only needs to focus on the "first time it creates the tags file", so it could offer interactive tools for the user to assemble a customized command line.

This is also a step towards incremental updating: https://github.com/universal-ctags/ctags/issues/2697

[AmaiKinono]

I found I can actually introduce "private ptag" to a tags file:

$ readtags -t tags -D
!_CITRE_PRIVATE_PTAG    value   /comment/
!_TAG_FILE_FORMAT       2       /extended format; --format=1 will not append ;" to lines/
!_TAG_FILE_SORTED       1       /0=unsorted, 1=sorted, 2=foldcase/
!_TAG_OUTPUT_EXCMD      mixed   /number, pattern, mixed, or combineV2/
...

I'll explore the concept of "command line arguments ptags" in Citre first.

Also, this means we can inject private ptags to tags file generated by other tools (kind of like the edittags program concept). As long as we use readtags to read it, they can be used.

[AmaiKinono]

I think I'd use a CITRE_TAGS_CMD ptag like:

ctags|-f|%thisfile%|--fields=*|-R|./src

| is the separator of arguments, %thisfile% is replaced by the path of the tags file. Of course We need to introduce escape sequences for using | and % inside the arguments.

In this way we can store the whole command line into one ptag.

This is just a preliminary thought.

[masatake]

I'm thinking about the same topic. Could you try #2741?

Instead of recording whole the command line as a string, I thought splitting it to parts semantically.

[yamato@control]~/var/ctags-github% ./ctags -o - --extras=+p main/main.c | grep '^!'
!_TAG_EXTRA_DESCRIPTION anonymous       /Include tags for non-named objects like lambda/
!_TAG_EXTRA_DESCRIPTION fileScope       /Include tags of file scope/
!_TAG_EXTRA_DESCRIPTION guest   /Include tags generated by guest parsers/
!_TAG_EXTRA_DESCRIPTION pseudo  /Include pseudo tags/
!_TAG_EXTRA_DESCRIPTION subparser       /Include tags generated by subparsers/
!_TAG_FIELD_DESCRIPTION epoch   /the last modified time of the input file (only for F\/file kind tag)/
!_TAG_FIELD_DESCRIPTION file    /File-restricted scoping/
!_TAG_FIELD_DESCRIPTION input   /input file/
!_TAG_FIELD_DESCRIPTION language        /Language of input file containing tag/
!_TAG_FIELD_DESCRIPTION name    /tag name/
!_TAG_FIELD_DESCRIPTION pattern /pattern/
!_TAG_FIELD_DESCRIPTION typeref /Type and name of a variable or typedef/
!_TAG_FILE_FORMAT       2       /extended format; --format=1 will not append ;" to lines/
!_TAG_FILE_SORTED       1       /0=unsorted, 1=sorted, 2=foldcase/
!_TAG_KIND_DESCRIPTION!C        U,unknown       /unknown objects/
!_TAG_KIND_DESCRIPTION!C        d,macro /macro definitions/
!_TAG_KIND_DESCRIPTION!C        e,enumerator    /enumerators (values inside an enumeration)/
!_TAG_KIND_DESCRIPTION!C        f,function      /function definitions/
!_TAG_KIND_DESCRIPTION!C        g,enum  /enumeration names/
!_TAG_KIND_DESCRIPTION!C        h,header        /included header files/
!_TAG_KIND_DESCRIPTION!C        m,member        /struct, and union members/
!_TAG_KIND_DESCRIPTION!C        s,struct        /structure names/
!_TAG_KIND_DESCRIPTION!C        t,typedef       /typedefs/
!_TAG_KIND_DESCRIPTION!C        u,union /union names/
!_TAG_KIND_DESCRIPTION!C        v,variable      /variable definitions/
!_TAG_OUTPUT_EXCMD      mixed   /number, pattern, mixed, or combineV2/
!_TAG_OUTPUT_FILESEP    slash   /slash or backslash/
!_TAG_OUTPUT_MODE       u-ctags /u-ctags or e-ctags/
!_TAG_PATTERN_LENGTH_LIMIT      96      /0 for no limit/
!_TAG_PROC_CWD  /home/yamato/var/ctags-github/  //
!_TAG_PROGRAM_AUTHOR    Universal Ctags Team    //
!_TAG_PROGRAM_NAME      Universal Ctags /Derived from Exuberant Ctags/
!_TAG_PROGRAM_URL       https://ctags.io/       /official site/
!_TAG_PROGRAM_VERSION   5.9.0   /61e526618/
!_TAG_ROLE_DESCRIPTION!C!header local   /local header/
!_TAG_ROLE_DESCRIPTION!C!header system  /system header/
!_TAG_ROLE_DESCRIPTION!C!macro  undef   /undefined/
!_TAG_ROLE_DESCRIPTION!C!member called  /indirectly calling a function via this member/
!_TAG_ROLE_DESCRIPTION!C!member initialized     /initialized/
!_TAG_ROLE_DESCRIPTION!C!member referenced      /referenced/
!_TAG_ROLE_DESCRIPTION!C!unknown        caseLabel       /case labels/

[AmaiKinono]

I'll try it later.

I understand why you want to use multiple ptags for command line args (to be able to compare them). But there are some advantages of using a single ptag in other ways:

• Tweaking the arguments are easy. Just edit the tags file and regenerate it.
• It's easier to inject and use for other tagging tools like hasktags, gotags.

I don't know if this is realistic, but how about using a single ptag, and when comparing 2 sets of command line args, calculate the information you need first (those *_DESCRIPTION info), then compare? .ctags.d/ may be a problem this way.

[AmaiKinono]

I tried #2741.

I think I know the idea of multi-pass parsing. I also re-read the example you gave before: https://github.com/universal-ctags/ctags/issues/2697#issuecomment-734969446

However, I haven't understand why multi-pass parsing needs ptags. From what I've seen, in the C example you gave in #2741, delete these lines:

!_TAG_FIELD_DESCRIPTION!C++	macrodef	/macro definition/
!_TAG_KIND_DESCRIPTION!C++	d,macro	/macro definitions/
!_TAG_ROLE_DESCRIPTION!C++!macro	undef	/undefined/

doesn't stop the macro expansion.

But I have other good news. I thought about a complete set of ptags for incremental updating. I'll show you how I built it step-by-step.

---

Step 1: re-generate a tags file

For this one, we need:

• Where is ctags run? (TAG_PROC_CWD)
• What are the command line arguments? The CITRE_TAGS_CMD ptag I mentioned above could do this. (If you use it you may want to rename it like TAGS_CMD or something alike)

When we re-generate the tags file, we cd to the CWD, and run the command line.

This is only for saving some typing when re-generate a tags file. The user could edit CITRE_TAGS_CMD or option files to tweak the options.

---

Step 2: Incremental updating

We've talked about this a lot in #2697.

The key is to ensure "reproducible on unchanged files". This means we need to find a way to use ctags so that it produces the same result on unchanged files (i.e., assuming the files are in the same state when this tags file was generated), then we use ctags in the same way for updated and newly added files.

To ensure reproducibility, we need to make sure:

0. TAG_PROC_CWD
1. command line arguments
2. option files
3. ctags program

are the same.

For 0, we let ctags cd to the cwd first, so they'll always be the same.

For 1, we could use 2 ptags:

!_CITRE_TAGS_CMD	ctags|-f|%thisfile%|--fields=*|-R|./src	/You can edit this/
!_CITRE_TAGS_LAST_CMD	ctags|-f|%thisfile%|--fields=*|-R|./src	/Don't edit this/

When the user edit CITRE_TAGS_CMD, we'll know by comparing it to CITRE_TAGS_LAST_CMD, so we do full updating (re-scan all files) and set CITRE_TAGS_LAST_CMD to the new command line.

For 2, we introduce !_TAG_OPTION_FILE_TIMESTAMP ptag. LIke:

!_TAG_OPTION_FILE_TIMESTAMP!/home/user/.ctags.d/0.tags	2020-11-14 04:55:32.965210497 +0900	/last modified time of option files/

When the set of option files used this time is not the same as recorded by these ptags, or the timestamp has changed, we do full updating.

For 3, we already have TAG_PROGRAM_NAME and TAG_PROGRAM_VERSION ptags.

When all the above equality tests are passed, we can do incremental updating confidently.

---

Step 3: TAG_**_DESCRIPTION

I still don't quite understand the use of these ptags. But I think, you don't have to give them the task of ensuring reproducibility, as reproducibility can be ensured by the above ptags.

Hope I've shedded some light on the problem.

Since Citre users ask me for a good tags file generating/updating tool, I'll explore step 1 in Citre, using private ptags.