Making reference tags for external entities instead of definitions tags

[masatake]

Some parsers capturing the tokens referring to external entities as definitions tags. This should not[1]. Instead, it should capture as reference tags. The modifications for solving this issue may break compatibility. However, implementing multi-pass parsing in the future, we must fix them now.

• [ ] using in C++
• [x] #include in C/C++
• [x] import in Python
• [x] \input and \include in Tex
• [x] require/require_relative/load in Ruby
• [ ] autoload in Ruby
• [x] use in Perl
• [ ] import in JavsScript
• [x] source and . in Shell
• [ ] `include in SystemVerilog
• [x] import in Protobuf
• [x] import in Go
• [x] include:: in ReStructuredText
• [ ] include::in asciidoc ...

---

[1] The following question illustrates the reason: https://stackoverflow.com/questions/3609433/skip-python-import-statements-in-exuberant-ctags

[masatake]

If external entities are captured with proper kinds and proper roles, we can implement semantic recursion feature.

I have implemented such feature as a prototype (-S option):

[yamato@slave ctags-github]$ u-ctags --kinds-python=F --sort=no -o - --extras=+r -S /usr/lib/python3.6/site-packages/pygments/cmdline.py  
u-ctags --kinds-python=F --sort=no -o - --extras=+r -S /usr/lib/python3.6/site-packages/pygments/cmdline.py  
__main__	/usr/lib/python3.6/site-packages/pygments/__init__.py	/^if __name__ == '__main__':  # pragma: no cover$/;"	m	language:PythonMain
codetagify	/usr/lib/python3.6/site-packages/pygments/filters/__init__.py	/^    'codetagify':     CodeTagFilter,$/;"	f	language:ansibleFilter	pyexpr:CodeTagFilter
keywordcase	/usr/lib/python3.6/site-packages/pygments/filters/__init__.py	/^    'keywordcase':    KeywordCaseFilter,$/;"	f	language:ansibleFilter	pyexpr:KeywordCaseFilter
highlight	/usr/lib/python3.6/site-packages/pygments/filters/__init__.py	/^    'highlight':      NameHighlightFilter,$/;"	f	language:ansibleFilter	pyexpr:NameHighlightFilter
raiseonerror	/usr/lib/python3.6/site-packages/pygments/filters/__init__.py	/^    'raiseonerror':   RaiseOnErrorTokenFilter,$/;"	f	language:ansibleFilter	pyexpr:RaiseOnErrorTokenFilter
whitespace	/usr/lib/python3.6/site-packages/pygments/filters/__init__.py	/^    'whitespace':     VisibleWhitespaceFilter,$/;"	f	language:ansibleFilter	pyexpr:VisibleWhitespaceFilter
gobble	/usr/lib/python3.6/site-packages/pygments/filters/__init__.py	/^    'gobble':         GobbleFilter,$/;"	f	language:ansibleFilter	pyexpr:GobbleFilter
tokenmerge	/usr/lib/python3.6/site-packages/pygments/filters/__init__.py	/^    'tokenmerge':     TokenMergeFilter,$/;"	f	language:ansibleFilter	pyexpr:TokenMergeFilter
__main__	/usr/lib/python3.6/site-packages/pygments/lexers/_mapping.py	/^if __name__ == '__main__':  # pragma: no cover$/;"	m	language:PythonMain
__main__	/usr/lib/python3.6/site-packages/pygments/formatters/_mapping.py	/^if __name__ == '__main__':  # pragma: no cover$/;"	m	language:PythonMain

As input file, only /usr/lib/python3.6/site-packages/pygments/cmdline.py is passed to ctags. The python parser of ctags caputres imported modules. ctags main part runs the python parser on the captured modules recursively when -S option is given. (To make the output smaller, only F kind entries are listed here.)

Combining multi pass parsing feature with this sematic recursion will be quite powerful; ctags can run C parser on a C source file with knowledges about all macro definitions used in the C source file.

[masatake]

An interesting question related to this issue https://stackoverflow.com/questions/65867522/create-tags-for-files-using-ctags

[AmaiKinono]

An interesting question related to this issue https://stackoverflow.com/questions/65867522/create-tags-for-files-using-ctags

It is an interesting one. I think at least it can be solved in 3 ways:

• Let the client tool recognize the user's intend and transform chapters/chapter1 into chapter1.tex for matching. I think this is easy to hack if the client tool they use doesn't utilize readtags.
• Let the client tool don't match by name, but by the input field, and filter those have file/F kind. This is easy with readtags.
• Let ctags generate reference tags, then the client tool needs to make sure it grabs chapters/chapter1 as the symbol.

[masatake]

ctags itself also must have an ability to solve file name from given language object name when implementing multi-pass parsing. See if (strcmp(suffix, ".py") != 0) in https://github.com/universal-ctags/ctags/pull/2741/commits/4aef85d631c000457b389c0d3c81c11d3ce6a2f1#diff-40f2d292b59e7d5e66dc889a6f66ba3bab41657faacd15b9f87f6b6ed848dfc3R1097 .

Let ctags generate reference tags, then the client tool needs to make sure it grabs chapters/chapter1 as the symbol.

It is already done in ctags side!

[jet@living]~/var/ctags-new% ./ctags --extras=+r --fields=+rE -o - /tmp/foo.tex 
chapters/chapter1	/tmp/foo.tex	/^\\include{chapters\/chapter1}$/;"	i	roles:included	extras:reference