squizlabs
Added documentation
Since this is an abstract class, it would be beneficial to document the expectations of the child class. Any function that returns an array should document the format of the array.
I agree that it would be a good thing™ for this array to be better documented.
In the PHPCS 2.x branch, the array was documented in the File class and, while that documentation may need an update, I think it would be a very good idea to bring it back.
See: https://github.com/squizlabs/PHP_CodeSniffer/blob/2.9/CodeSniffer/File.php#L26-L106
I think it would be a very good idea to bring it back
Agreed. I think it's mostly just the same, but probably with a few more indexes. It probably belongs in the wiki docs as it is easier to format and manage there, but with a link off the the docs from the code.
In reading the code for JS and PHP, I believe this is the expected result of the tokenize function. If other indexes are added to the tokens index, they happen at later stages. If tokens were their own object, it would be more obvious where to document the structure.
While the documentation added here is strictly correct, IMO it is misleading as the $tokens array is enhanced with numerous other keys after this initial run and those other keys would now not be documented.
IMO, either the documentation in the File class docblock as was available in PHPCS2.x should be brought back & updated to reflect the current keys, or as @gsherwood suggested, this documentation should be added to the wiki.
I'm marking this as a close candidate as I don't think merging this as-is has added value.