ctags
ctags copied to clipboard
universal-ctags
JavaScript: SAPUI5 / OpenUI5 support
The javascript parser supports SAP's UI5 framework, which was also open sourced as OpenUI5 (.
I added the initial support for it in Exuberant tags and I work with it daily. I have updated the support for it in universal tags. I will include a pull request for my changes. Fixed a few other issues along the way while I was testing.
I do have some questions about adding tests and what not. I see a bunch of directories under the Units directory. Could someone provide me some instruction on how to use these directories. First I would like to run all the javascript tests to make sure it is has no regressions.
I would also like to understand how to add my own set of tests for UI5.
Thank you. David
After getting your pull request. I would like you to allow me to rearrange your code. I would like to introduce language named OpenUI5 based on your pull request.
Backgroupd:
Some of languages has "dialects". For example taken from your issue, Javascript language has OpenUI5 as a dialect. Another example is in #1227; C++ has CUDA as a dialect. Python has CPython. M4 has autoconf. Make has Automake....
I wonder how ctags deals dialects. The classical way is just extending kinds and file name patterns of the base language. #1220 is typical case. However, this approach has an issue. This approach consumes more kind letters. A namespace of kind letters are not small[a-zA-Z]. However, it is not enough large for adding many dialects. Not only us but also users will add dialects locally, soon we will see conflicts of kind letters.
Look at https://github.com/universal-ctags/ctags/issues/453 One wants to add 'c' (constant) kind to Ruby. However, it is used already for Rspec dialect.
From these experience I think dialect should has its own parser derived from the base language. The dialect has its own namespace. The namespace of the base language can be kept clean. Further more language: field can show the name of dialect.
How can we implement a dialect? I have tired to answer this question for long time. I have developed some facilities. I have to explain them.
I misunderstood your question. To run test, do
$ make units
Only about javascript do
$ make units LANGUAGES=JavaScript
How to write a test case, please read http://docs.ctags.io/en/latest/units.html .
After thinking I rename what I called "dialect" here to "subparser". See #1326, especially https://github.com/universal-ctags/ctags/commit/9ac9b7a0f666234dcb1eff6486f6b0fd93d6d299 . In this case, javascript parser acts as a base parser. openui5 parser can be implemented as a subparser based on the base parser.
I convinced that UI5 code should be parsed by a subparser of JavaScript.
sap.ui.controller("id.of.controller", {
method_name1 : function... {
},
method_name2 : function ... {
}
}
method_name1 and method_name2 are captured well with the JavaScript parser. However, to fill the scope fields of them, "id.of.controller" must be captured.
The question is what "kind" we should assign to "id.of.controller"? The kind should be part of UI5 subparser. The method_names may be tagged two times, one time by the JavaScript parser. the other time by UI5 parser. @dfishburn, do you have any comment?
@b4n, I'm working on introducing the corkAPI to the JavaScript parser to support "export and "import" modules to support Vue files.
@b4n, I'm working on introducing the corkAPI to the JavaScript parser to support "export and "import" modules to support Vue files.
Not sure what that means, why do we need cork to parse import and export syntax? Using cork is nice anyway, but I'm not sure I understand why it's needed here, but I'm probably missing something.
Anyway that's nice :) Though, I'm afraid it won't be a trivial task, because the JS parser has its share of complex situations, and often emits parent tags after children (mostly to decide and type & co, so that can possibly be amended after initial emission). Good luck, and although I'm short on spare time lately (sorry :confused:) I'll try and look at anything you might need here (I know I have other PRs I need to look at, just give me a little time :slightly_smiling_face: ).
and often emits parent tags after children
That's the point! Here is my solution.
commit 02a3804502981e2574dd0f64d47011d76390c3e7
Author: Masatake YAMATO <[email protected]>
Date: Thu Jun 13 04:37:54 2019 +0900
main: introduce a function putting a placeholder tag in the corkQueue
Consider following input written in imaginary programming language X:
A {
B;
}
B have A as its parent scope.
With using the cork API, you can make tags with the following step:
1. make the tag for A, and get the cork index (index_A) for A, and
2. make the tag for B filling scopeIndex field with index_A.
In the most of all case, these steps work well.
However, there is an exception. The language X has following odd
character; the kind of A can be decided after parsing B. For such
language character, above steps don't work.
You cannot make the tag for A before making B because the kind for A
cannot be decided till parsing B. However, you cannot make the tag for
B before making the tag for A because index_A is needed to make the
tag for B. Mutual dependencies are there.
This change introduces makePlaceholder function. This function extends
the corkAPI to work well even if the mutual dependencies are there.
makePlaceholder() allows to make a temporary tag (placeholder).
Reconsider the language X. When the parser finds A, the parser can
call makePlaceholder(A). The parser can make a tag for A without
specifying its kind. It returns the corkIndex for the placeholder. The
parser can use it index_A when parsing and making tag for B. The
parser gets the cork index (index_B) for B.
After parsing B, now the parser knows the real kind for A. The parser
can convert the placeholder tag to the real tag for A by patching the
placeholder tag; the parser refills the kindIndex field of A with the
real kind.
Signed-off-by: Masatake YAMATO <[email protected]>
Newly introduced makePlaceholder() and its variant allows me to write following code:
+static void patchScopeOfEntries (int start, int end)
+{
+ if (! (end - start > 1))
+ return;
+
+ for (int i = start + 1; i < end; i++)
+ {
+ tagEntryInfo *e = getEntryInCorkQueue (i);
+ if (e && e->extensionFields.scopeIndex == start)
+ e->extensionFields.scopeIndex = end;
+ }
}
...
@ -2190,11 +2222,14 @@ nextVar:
goto cleanUp;
}
- is_class = parseBlock (token, name->string);
+ int p = makeSimplePlaceholder (name->string), q;
+ is_class = parseBlock (token, p);
if ( is_class )
- makeClassTag (name, signature, NULL);
+ q = makeClassTag (name, signature, NULL);
else
- makeFunctionTag (name, signature, is_generator);
+ q = makeFunctionTag (name, signature, is_generator);
+ patchScopeOfEntries (p, q);
+ indexForName = q;
if ( vStringLength(secondary_name->string) > 0 )
makeFunctionTag (secondary_name, signature, is
I'd like to show my current idea about modules. I wonder you may think it is over-engineering:-) Please, read when you have a time.
To support vue, we have to think of module.
1: var x = { methodA: function () {} };
2: export x;
When thinking of modules, a user of ctags may want x in both L1 and L2 to be tagged.
x in the line 2 may be captured as a reference tag with role "exported".
However, what "kind" should be assigned to the x in the line 2?
The kind should be "variable" because it is defined as a variable in the line 1.
For assinging "variable" kind to "x" in line 2, the JavaScript parser must be able to remember the definition tag for "x". In other words, we need a symbol table.
I think the symbol table can be built on the corkQueue.
Before designing the symbol table, we have to make the JavsScript parser use the corkAPI.
I assume we will have the cork based JavaScript parser. Based on the assumption, I designed how modules, named exported and/or imported from the modules can be captured in consistent way. I will paste my note.
.. _javascript:
======================================================================
The JavaScript parser
======================================================================
Making tags for modules
---------------------------------------------------------------------
(This section is written by @masatake).
To tag names in "export" and "import" statements, following kinds, roles, and
fields are introduced.
Kinds
"name"
This kind mainly is used for N of "X as N" form. When an
language object X is imported or exported with the different
name N, the JavaScript parser assigns "name" kind to N.
A "name" kind tag may have "exprots" or "imports" field. The
fields specifies X of "X as N" form. In other word, the fields
specifies the real language object exported with the name N.
"data"
This kind represents something unknown data. e.g. when the
JavaScript parser read a line, "import X from M;", the
parser cannot know whether X is a variable, a constant, a class
or a function because X is defined in M. In such case, the parser
assigns the "data" kind to X as a placeholder.
This kind mainly used for reference tags. If the parser finds
definition for a "data", the parser can assign more specific
kind instead.
Depending on the statement where X is appeared, "data" kind tag
takes one of three roles: "exported", "imported", or "passthrough".
Depending on the statement where X is appeared, "data" kind tag may have
some of three fields: "exportedAs", "importedAs", and/or "origin".
"exportedAs" and "importedAs" fields are attached when tagging
X of "X as N" form. "exportedAs:N" is attached to the "data" tag
of X if the form is in a "export" statement. "importedAs:N" is attached
to the "data" tag of X if the form is in a "import" statment.
If X appears in "X as N from M" from, "origin:M" field is attached to
X. Here M represents a module.
"module"
As name shown, it represents a referenced module.
If the module name is appeared in an "export" statement,
the parser assigns "aggregated" role to the "module" kind tag.
If the module name is appeared in an "import" statement,
the parser assigns "imported" role to the "module" kind tag.
Roles
exported
imported
passthrough
aggregated
Fields
exportedAs
exports
origin
importedAs
imports
Handling "export" statement
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This section explains how the JavaScript parser tags names defined
and/or referenced in export statements.
See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/export
about export statements.
export statements have following forms (quoted from the page):
.. code-block:: JavaScript
// Exporting individual features
export let name1, name2, ..., nameN; // also var, const
export let name1 = ..., name2 = ..., ..., nameN; // also var, const
export function functionName(){...}
export class ClassName {...}
// Export list
export { name1, name2, ..., nameN };
// Renaming exports
export { variable1 as name1, variable2 as name2, ..., nameN };
// Default exports
export default expression;
export default function (...) { ... } // also class, function*
export default function name1(...) { ... } // also class, function*
export { name1 as default, ... };
// Aggregating modules
export * from ...;
export { name1, name2, ..., nameN } from ...;
export { import1 as name1, import2 as name2, ..., nameN } from ...;
export { default } from ...;
Exporting individual features
,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Input:
.. code-block:: JavaScript
I1: export let name1, name2, ..., nameN; // also var, const
I2: export let nameA = ..., nameB = ..., ..., nameZ; // also var, const
I3: export function functionName(){...}
I4: export class ClassName {...}
Expected tags:
.. code-block:: tags
T1: name1 input.mjs /^...$/;" line:I1 kind:variable roles:def,exported
...
T2: nameA input.mjs /^...$/;" line:I2 kind:variable roles:def,exported
...
T3: functionName input.mjs /^...$/;" line:I3 kind:function roles:def,exported
T4: className input.mjs /^...$/;" line:I4 kind:class roles:def,exported
These statements do two things at once: defining names and exporting
them. The JavaScript parser makes a tag having both "def" and
"exported" roles for each name.
Export list
,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Input:
.. code-block:: JavaScript
// No definition for name2,... in the input.
I1: var name1;
...
I2: export { name1, name2, ..., nameN };
Expected tags:
.. code-block:: tags
T1: name1 input.mjs /^...$/;" line:I1 kind:variable roles:def
...
T2: name1 input.mjs /^...$/;" line:I2 kind:variable roles:exported
T3: name2 input.mjs /^...$/;" line:I2 kind:data roles:exported
...
This statement just refers names. The JavaScript parser emits
reference tags for the names. The issue is what kind should be
assigned to the tags. If the names are defined in the input (I1), the
JavaScript parser captures them (T1) before parsing the export
statement. In this case, the parser can assign correct kinds for the
associated reference tags (T2).
If the parser cannot find some of the names in the input, the parser
assigns newly introduced "data" kind to the reference tags for the
names as placeholder. See T3, name2 has "data" kind because name2 is
not defined in the Input.
Renaming exports
,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Input:
.. code-block:: JavaScript
// No definition for variable2,... in the input.
I1: var variable1;
I2: export { variable1 as name1, variable2 as name2, ..., nameN };
Expected tags:
.. code-block:: tags
T1: variable1 input.mjs /^...$/;" line:I1 kind:variable roles:def
T2: variable1 input.mjs /^...$/;" line:I2 kind:variable roles:exported exportedAs:name1
T3: name1 input.mjs /^...$/;" line:I2 kind:name roles:def exports:variable1
T4: variable2 input.mjs /^...$/;" line:I2 kind:data roles:exported exportedAs:name2
T5: name2 input.mjs /^...$/;" line:I2 kind:name roles:def exports:variable2
In the form "V as N", V is indirectly exported with name N. For the
form, the JavaScript parser makes a reference tag for V, and a
definition tag for N.
The way to assign a kind for V is the same as explained in "Export
list". For N, newly introduced "name" kind is assigned.
Newly introduced fields, "exports" and "exportedAs", are for
representing the relation between V and N. "exports" field attached
to N represents what N exports. "exportedAs" field attached to V
represents the name with which V is exported.
Default exports
,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Input:
.. code-block:: JavaScript
I1: export default identifier; // No definition for identifier in the input file
I2: export default expression;
I3: export default function (...) { ... } // also class, function*
I4: export default function name1(...) { ... } // also class, function*
I5: var name2; export { name2 as default, ... };
I6: export { name3 as default, ... }; // No definition for name3 in input file
Expected tags:
.. code-block:: tags
T1: identifier input.mjs /^...$/;" line:I1 kind:data roles:exported exportedAs:default
T2: default input.mjs /^...$/;" line:I1 kind:name roles:def exports:identifier
T3: AnonymousTag0 input.mjs /^...$/;" line:I2 kind:data roles:def,exported exportedAs:default
T4: default input.mjs /^...$/;" line:I2 kind:name roles:def exports:AnonymousTag0
T5: AnonymousTag1 input.mjs /^...$/;" line:I3 kind:function roles:def,exported exportedAs:default
T6: default input.mjs /^...$/;" line:I3 kind:name roles:def exports:AnonymousTag1
T7: name1 input.mjs /^...$/;" line:I4 kind:function roles:def,exported exportedAs:default
T8: default input.mjs /^...$/;" line:I4 kind:name roles:def exports:name1
T9: name2 input.mjs /^...$/;" line:I5 kind:variable roles:def
T10: name2 input.mjs /^...$/;" line:I5 kind:variable roles:exported exportedAs:default
T11: default input.mjs /^...$/;" line:I5 kind:name roles:def exports:name2
T12: name3 input.mjs /^...$/;" line:I6 kind:data roles:exported exportedAs:default
T13: default input.mjs /^...$/;" line:I6 kind:name roles:def exports:name3
"export default" form allows exporting a language object without naming.
If the language object is an identifier (I1), the JavaScript parser
tags it with "exported" role (T1). In addition, the parser attaches
"exportedAs:default" field to the tag. If the definition for the name is
in the input file (I5), the kind for the definition tag of the name is
assigned to the reference having the same name. If the definition
for the name is not in the input file (I1), "data" kind is assigned to
the reference tag for the name (T1).
If the language object is not an identifier but has a name (I4), the
name is used for making a tag. The roles for the tag are "def" and
"exported" (T7). "exportedAs:default" field is also attached to the
tag.
If the language object is not an identifier and doesn't have a name
(I2, I3), anonymous name generated by ctags is used as name for
tagging (T3, T5).
In any forms in "Default exports", the JavaScript parser emits
"default" with "name" kind for each export statement. Strictly
speaking, "default" is a keyword of JavaScript language, not a
defined name. However, the tag for "default" may help a client
tool find exported language objects from a input file quickly.
A "default" tag has "exports" field representing what language
object indirectly exported via the "default" keyword.
Aggregating modules
,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Input:
.. code-block:: JavaScript
I1: export * from MJS0;
I2: export { name1, name2, ..., nameN } from MJS1;
I3: export { import1 as nameA, import2 as nameB, ..., nameZ } from MJS2;
I4: export { importX as default } from MJS3;
I5: export { default } from MJS4;
Expected tags:
.. code-block:: tags
T1: MJS0 input.mjs /^...$/;" line:I1 kind:module roles:aggregated
T2: * input.mjs /^...$/;" line:I1 kind:data roles:passthrough origin:MJS0
T3: MJS1 input.mjs /^...$/;" line:I2 kind:module roles:aggregated
T4: name1 input.mjs /^...$/;" line:I2 kind:data roles:passthrough origin:MJS1
T5: MJS2 input.mjs /^...$/;" line:I3 kind:module roles:aggregated
T6: import1 input.mjs /^...$/;" line:I3 kind:data roles:passthrough origin:MJS2 exportedAs:name1
T7: nameA input.mjs /^...$/;" line:I3 kind:name roles:def exports:import1
T8: MJS3 input.mjs /^...$/;" line:I4 kind:module roles:aggregated
T9: importX input.mjs /^...$/;" line:I4 kind:data roles:passthrough origin:MJS3 exportedAs:default
T10: default input.mjs /^...$/;" line:I4 kind:name roles:def exports:importX
T11: MJS4 input.mjs /^...$/;" line:I5 kind:module roles:aggregated
???
T12: AnonymousTag0 input.mjs /^...$/;" line:I5 kind:data roles:passthrough origin:MJS4 exportedAs:default
T13: default input.mjs /^...$/;" line:I5 kind:name roles:def exports:AnonymousTag0
or
T12: default input.mjs /^...$/;" line:I5 kind:name roles:passthrough origin:MJS4 exports:default
or
T12: default input.mjs /^...$/;" line:I5 kind:data roles:passthrough origin:MJS4 exportedAs:default
T13: default input.mjs /^...$/;" line:I5 kind:name roles:def exports:default
or
T12: default input.mjs /^...$/;" line:I5 kind:name roles:def,passthrough exports:default origin:MJS4 exportedAs:default
In the forms, language objects exported from a module are exported
from the input. "passthrough" role is introduced for representing the
language objects. Combining "imported" and "exported" roles cannot
represent what "passthrough" role represents because the language
objects exported from the external module are not imported to the
input.
"module" kind is introduced to represent the language objects
specified in "from ..." form of export statement. The "module" kind is
also used to represent the language objects specified in import
statements. "aggregated" role is for representing "from ..." form.
"origin" field is attached to tags having "passthrough" role. It
represents the module where the language object comes from.
The kind for passthrough'ed language objects is "data" because the
definitions for the language objects may be in the external module,
not in the input.
Handling "import" statment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This section explains how the JavaScript parser tags names defined
and/or referenced in import statements.
See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import
about import statements.
import statements have following forms (quoted from the page):
.. code-block:: JavaScript
I1: import defaultExport0 from "module-name1";
I2: import * as name0 from "module-name2";
I3: import { export0 } from "module-name3";
I4: import { export1 as alias1 } from "module-name4";
I5: import { export2 , export3 } from "module-name5";
I6: import { export4 , export5 as alias2 , [...] } from "module-name6";
I7: import defaultExport1, { export6 [ , [...] ] } from "module-name7";
I8: import defaultExport2, * as name1 from "module-name8";
I9: import "module-name9";
I10: var promise = import(module-name10);
.. code-block:: tags
T1: module-name1 input.mjs /^...$/;" line:I1 kind:module roles:imported
???
T2: default input.mjs /^...$/;" line:I1 kind:data roles:imported importedAs:defaultExport origin:module-name1
T3: defaultExport0 input.mjs /^...$/;" line:I1 kind:name roles:def imports:default
or
T2: defaultExport0 input.mjs /^...$/;" line:I1 kind:name roles:def imports:default origin:module-name1
T4: module-name2 input.mjs /^...$/;" line:I2 kind:module roles:imported
T4: * input.mjs /^...$/;" line:I2 kind:data roles:imported importedAs:name0 origin:module-name2
T5: name0 input.mjs /^...$/;" line:I2 kind:name roles:def imports:*
T6: module-name3 input.mjs /^...$/;" line:I3 kind:module roles:imported
T7: export0 input.mjs /^...$/;" line:I3 kind:data roles:imported origin:module-name3
T8: module-name4 input.mjs /^...$/;" line:I4 kind:module roles:imported
T9: export1 input.mjs /^...$/;" line:I4 kind:data roles:imported importedAs:alias1 origin:module-name4
T10: alias1 input.mjs /^...$/;" line:I4 kind:name roles:def imports:export1
T11: module-name5 input.mjs /^...$/;" line:I5 kind:module roles:imported
T12: export2 input.mjs /^...$/;" line:I5 kind:data roles:imported origin:module-name5
T13: export3 input.mjs /^...$/;" line:I5 kind:data roles:imported origin:module-name5
T14: module-name6 input.mjs /^...$/;" line:I6 kind:module roles:imported
T15: export4 input.mjs /^...$/;" line:I6 kind:data roles:imported origin:module-name6
T16: export5 input.mjs /^...$/;" line:I6 kind:data roles:imported importedAs:alias2 origin:module-name6
T17: alias2 input.mjs /^...$/;" line:I6 kind:name roles:def imports:export5
T18: module-name7 input.mjs /^...$/;" line:I7 kind:module roles:imported
T19: defaultExport1 input.mjs /^...$/;" line:I7 kind:name roles:def imports:default origin:module-name7
T20: export6 input.mjs /^...$/;" line:I7 kind:data roles:imported origin:module-name7
T21: module-name8 input.mjs /^...$/;" line:I8 kind:module roles:imported
T22: defaultExport2 input.mjs /^...$/;" line:I8 kind:name roles:def imports:default origin:module-name8
T23: * input.mjs /^...$/;" line:I8 kind:data roles:imported importedAs:name1 origin:module-name8
T24: name1 input.mjs /^...$/;" line:I8 kind:name roles:def imports:*
T25: module-name9 input.mjs /^...$/;" line:I8 kind:module roles:imported
Two use cases are considered when designing how JavaScript tags the
names in the statements.
Jump to definition
A JavaScript source file X imports module Y. A name y exported
from Y is used in X. A user of ctags reading X wants to know the
definition of y. The JavaScript of ctags should provide following
tags for support this use case:
the definition tag for y if y is exported directly, or the
definition tag for y that has exprots:z field if z is exported
indirectly.
In the later case, one more step is needed to jump the place where
y points. Find the definition tag for z.
List all names exported from a module
Symbol completion feture may need this.
The names are tagged as reference tags with exported role.