sdk icon indicating copy to clipboard operation
sdk copied to clipboard

uast.TokenOf returns unexpected results for position nodes

Open bzz opened this issue 6 years ago • 0 comments

As a user, I expect to be able to extract all the tokens from UAST using helpers from #363 with something like:

	var tokens []string
	iter := tools.NewIterator(root, tools.PreOrder)
	for n := range tools.Iterate(iter) {
		token := uast.TokenOf(n)
		if token != "" {
			tokens = append(tokens, token)
		}
	}

and then by printing all the tokens get a "tokenized" version of particular file.

But uast.TokenOf returns non-empty results e.g for position nodes, with is although documents, is not very intuitive.

bzz avatar Jun 04 '19 16:06 bzz