Track positions of tokens

[sjoerdjob]

In some cases (error reporting, syntax highlighting), it might be desired to know the location of the tokens w.r.t. the original input.

[coveralls]

Coverage decreased (-0.06%) to 93.333% when pulling 3514252c5be9d7c6ada40784c5ba196299b722b4 on sjoerdjob:track_positions into 2936f9756aaea2e0f5b4cbb578a25c4fe07bcdf1 on andialbrecht:master.

[vmuriart]

would it be simpler to build the token during the lexing process and assign the row/col values then? then you are only yielding/refering 1 object instead of 4 related ones

[sjoerdjob]

Yes. That would be a lot simpler. I was just trying to keep the diff as simple as possible.

[vmuriart]

I was going to change into that behavior in the near future. Unless there is any particular reason not to, I think it would be better to go ahead with it. But you are correct, it would increase the diff since there would be changes needed in other section of the code simply for that change.

We can always go back and do it later though. Haven't fully reviewed this code yet.

[sjoerdjob]

Yes, going ahead with moving to tokens earlier would be a wise move. It's a bit confusing when it's uncertain if you're iterating over tuples or over Tokens.

[vmuriart]

Haven't forgotten about this. I'll merge it in after the 0.2.0 release.

[segoon]

Hi! Any update on this? I'd want to use the feature of line:col.

I see that there is a merge conflict here. If there is no developer to update the patch, I can try.

[sjoerdjob]

@segoon : I myself no longer have any interest in carrying this patch forward, so you're free to continue where this left off.