gh-120317: Lock around global state in the tokenize module

[lysnikolaou]

• Issue: gh-120317

[Fidget-Spinner]

This LGTM. Can you add the reproducer to test.test_free_threading please? I'll approve after!

[lysnikolaou]

@Fidget-Spinner I added more locking around global state. Reviewing each commit seprately might make it easier.

[lysnikolaou]

Feedback addressed.

@Fidget-Spinner I used _Py_atomic_store_int instead of the macro. Is that okay?

[lysnikolaou]

All feedback addressed. Thanks a lot for the reviews and the help @Fidget-Spinner and @erlend-aasland!

[pablogsal]

Actually, one question here: shouldn't we be protecting all reads of the it structure that are not under critical sections? Otherwise we are getting some partial reads no? I am missing something?

[pablogsal]

For example, this read is not protected, no?:

    Py_ssize_t lineno = ISSTRINGLIT(type) ? it->tok->first_lineno : it->tok->lineno;
     Py_ssize_t end_lineno = it->tok->lineno;

[lysnikolaou]

After some offline discussion with @pablogsal, we decided it makes more sense to just lock around all of tokenizeriter_next.

[lysnikolaou]

@pablogsal Can you do a final review here when you have some spare cycles?

[miss-islington-app[bot]]

Thanks @lysnikolaou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13. 🐍🍒⛏🤖

[bedevere-app[bot]]

GH-121841 is a backport of this pull request to the 3.13 branch.