Allow to customize some behaviors of Lexer, so that Extension instances can know raw block begin and end info.

[frankdu]

Problem: Currently, the "data" tokens don't have the information about if the "data" is inside the raw/endraw block, or not. This limit the Extension object's ability to treat the "data" tokens differently. I'm developing a jinja extension which is limited by this.

Feature Request:

Allow Lexer to optionally choose to emit the raw_begin and raw_end tokens, v.s. ignore them (current behaviors) Allow Environment to switch the lexer provider, so that you can inject your customized Lexer instances. For example, a customized one when an Extension is added, or when it's in testing.

Fixes

1. Allow token_begin and token_end in stream to enable Extensions to treat "data" tokens differently
2. Add the lexer_provider attrib to Environment, so to allow provide custom lexer in different use cases (e.g. testing)

Test Plan:

1. Run pytest

• fixes #1962

Checklist:

• [x] Add tests that demonstrate the correct behavior of the change. Tests should fail without the change.
• [x] Add or update relevant docs, in the docs folder and in code.
• [x] Add an entry in CHANGES.rst summarizing the change and linking to the issue.
• [x] Add .. versionchanged:: entries in any relevant code docs.
• [x] Run pre-commit hooks and fix any issues.
• [x] Run pytest and tox, no tests failed.