trace/profile conflict with the use of sys.modules[__name__]

[abalkin]

BPO	9914
Nosy	@birkenfeld, @ncoghlan, @abalkin, @ezio-melotti, @florentx, @ericsnowcurrently, @serhiy-storchaka, @applio
Files	issue5758_trace_execute_other_modules_main_v0.patch issue24676.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/abalkin'
closed_at = None
created_at = <Date 2010-09-21.18:17:43.355>
labels = ['type-bug', 'library']
title = 'trace/profile conflict with the use of sys.modules[__name__]'
updated_at = <Date 2017-01-30.14:08:10.444>
user = 'https://github.com/abalkin'

bugs.python.org fields:

activity = <Date 2017-01-30.14:08:10.444>
actor = 'davin'
assignee = 'belopolsky'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2010-09-21.18:17:43.355>
creator = 'belopolsky'
dependencies = []
files = ['26451', '39963']
hgrepos = []
issue_num = 9914
keywords = ['patch']
message_count = 6.0
messages = ['117090', '165928', '166466', '247014', '286484', '286491']
nosy_count = 10.0
nosy_names = ['georg.brandl', 'ncoghlan', 'belopolsky', 'ezio.melotti', 'flox', 'eric.snow', 'serhiy.storchaka', 'tati_alchueyr', 'davin', 'bli']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9914'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5', 'Python 3.6']

[abalkin]

The main() method of trace and profile modules attempt to emulate the environment in which traced code runs when invoked directly, but it fails in several respects. The specific problem which is the subject of this issue is that while __name__ is set to '__main__' in code globals, sys.modules['__main__'] still point to the trace/profile module.

Among other problems, this conflicts, with a popular idiom used in regression test scripts:

   support.run_unittest(__name__)

For example,

$ python -m trace -c -C trace.d Lib/test/test_optparse.py 


----------------------------------------------------------------------
Ran 0 tests in 0.001s

OK

No tests are ran because run_unittests() looks for test case classes in the trace module and finds none.

This is related to bpo-9323, so I am merging in the nosy list. See also r83393.

[19cea345-8b82-4375-b09b-591195212f03]

Yesterday I've studied this problem with flavio.ribeiro, and we've started "solving" it. The result of our progress is available at: issue5758_trace_execute_other_modules_main_v0.patch

The problem of our approach is that any code outside the condition "if __name__ == '__main__'" will be run twice, as we used imp.load_source to obtain trace's analyzed code and redefine sys.modules['__main__'].

In order to provide a better solution, we were considering using lazy module import, e.g: http://code.activestate.com/recipes/473888-lazy-module-imports/

Any thoughts on this?

[ncoghlan]

This is a tricky one. Long term, the right approach is to migrate all the "scripts that run other scripts" over to runpy, but the runpy API needs work before we can do that (see bpo-9325).

For bug fix purposes though, these modules can borrow some of runpy's infrastructure in order to fake the system state correctly. Specifically, the runpy._TempModule and runpy._ModifiedArgv0 context managers. (_TempModule may need a tweak to allow the module to be used to be passed in rather than always being implicitly created)

See runpy._run_module_code for an example of how to use them.

[serhiy-storchaka]

Yet one problem (with pickle) was reported in bpo-24676.

[b057632e-7e51-4152-bcda-de409e820509]

Just to report that this bug seems responsible for failures at cProfiling some code using multiprocessing:

http://stackoverflow.com/q/41892297/1878788 http://stackoverflow.com/q/11512499/1878788

[applio]

Though this issue is specifically concerned with runpy APIs and their impact especially in running unittest test scripts, it's worth commenting here for people who need a workaround in the short term: code such as that shared in http://stackoverflow.com/q/41892297/1878788 can be made to run happily by creating a second script which imports the first and simply runs the test(s) from there.

In the specific case of the 'forkiter.py' from http://stackoverflow.com/q/41892297/1878788, one would create a 'run_my_tests.py' with the contents:

from forkiter import main

if __name__ == "__main__":
    exit(main())

Now this invocation of cProfile runs happily because pickle is able to see the module where all the needed classes/functions were defined: python3.6 -m cProfile -o forkiter.prof ./run_my_tests.py

[Kentzo]

FWIW still happens with 3.10.8