1. Added IterateTokens class which can be used to iterate tokens as C++11 range-based loops.

for (const auto &tok : IterateTokens(scope)) {
  // ...
}

2. Added macro ITERATE_TOKENS(tok, ...), which generates regular for-begin-end loop:

for (ITERATE_TOKENS(tok, start, end)) { ... }
for (ITERATE_TOKENS(tok, scope)) { ... }
for (ITERATE_TOKENS(tok, mTokenizer)) { ... }

There was already tokenrange.h, which contained an iterator draft, but it wasn't used anywhere, and also it didn't support token assignment inside loops, so I removed it and replaced with tokeniterators.h.

Usage:

Start to end

Before:

for (const Token *tok = start; tok && tok != end; tok = tok->next()) {

After:

for (const auto &tok : IterateTokens(start, end)) {

or

for (ITERATE_TOKENS(tok, start, end)) {

Scopes

Before:

for (const Token* tok = scope->bodyStart->next(); tok != scope->bodyEnd; tok = tok->next()) {

After:

for (const auto &tok : IterateTokens(scope)) {

OR

for (const auto &tok : IterateTokens::ScopeIncludeBraces(f.functionScope))

OR

for (ITERATE_TOKENS(tok, scope)) { ... }

Tokenizer

Before:

for (const Token *tok = mTokenizer->tokens(); tok; tok = tok->next()) {

After:

for (const auto &tok : IterateTokens(mTokenizer)) {

OR

for (ITERATE_TOKENS(tok, mTokenizer)) { ... }

IterateTokens::UntilLinked

Before:

for (const Token *operTok = operStart; operTok != operStart->link(); operTok = operTok->next()) {

After:

for (const auto &tok : IterateTokens::UntilLinked(operStart)) {

Handles "jumping" correctly (assignment of token inside loop body):

This looks weird to me. It's not consistent with normal range for loop behavior. So I wonder if this handling can be removed.

I agree it's not consistent and I don't think it would be obvious to most developers what is being done.

Removed that feature (made assignment forbidden) and added ITERATE_TOKENS(tok, ...) macro which expands into regular for-begin-end loop and could replace existing loops without any impact.

I think this way is very good. Cause there are many different ways this is written in the code. And it actually introduces possible 'buffer' overflows. I just wonder if the check for 'end of range' should be like in precedes() defined in lib/astutils.h.

Also a small note, could we make precedes() inline? It actually make a noticeable difference. And if we start using this c++11-ish style of for loop it should be inlined.