asttokens
asttokens copied to clipboard
Strange behavior with tokens of import nodes
Hi!
I tried the following code to parse import statements :
source = 'import asttokens as at'
atok = asttokens.ASTTokens(source, parse=True)
I get the following AST:
Module
Import
alias
Which is OK. My purpose is to gather the position of the identifiers asttokens
and at
in the code. Therefore I was trying to get the tokens composing the alias node, which sounds logic because asttokens as at
is and alias definition. However when I ask for the tokens composing the alias node, using list(atok.get_tokens(atok.tree.body[0].names[0]))
I get the following result :
[Token(type=1, string=u'import', start=(1, 0), end=(1, 6), line=u'import asttokens as at', index=0, startpos=0, endpos=6)]
Which is was not what I was expecting, I was expecting to have the ['asttokens', 'as', 'at']
tokens.
Is there something I am missing?
Best regards.
You are right, the built-in ast
module doesn't seem to annotate the "alias" node with any useful information, so it doesn't have any location info. One reasonable thing we could do is add a method to mark_tokens.py
like:
def visit_alias(self, node, first_token, last_token):
asname = self._code.find_token(first_token, token.NAME, node.asname)
return (asname, asname)
This would map the "alias" node to the token with the alias name ("at" in your case). If you experiment with this and feel this is a useful change, a test and a pull request would be appreciated. Note also that there are somewhat harder situations, like from os import path as _path, wait as _wait
.
Hi!
Thanks for the answer!
I will try to cope with this problem, if I manage to do something, I will send a PR!
Cheers.