onelinerizer
onelinerizer copied to clipboard
csvoss
ENH: Implement support for with-blocks.
I missed the presentation tonight at Boston Python, but a colleague pointed me at this repository.
This PR is a work in progress toward support for with-blocks.
The basic idea here is rewrite the ast of the with-block into two statements and a try-except, which we already know how to oneline-ify
In general, we can rewrite a with block of the form:
with <expr> as <ctxname>:
<body>
as
__anonymous = <expr>
<ctxname> = __anonymous.__enter__()
try:
<body>
except:
if not __anonymous.__exit__(*sys.exc_info()):
raise
else:
__anonymous.__exit__(None, None, None)
Once we've rewritten the AST, we should be able to just delegate to existing the existing functionality for sequencing statements and handling try-except blocks.
This mostly works as-is, but for reasons I don't quite understand, statements after the with-block aren't being executed. I suspect that I'm not correctly doing whatever work is needed to sequence the with-block with subsequent statements. Any guidance here would be appreciated.
Thanks for looking at this.
Sequencing subsequent statements is the purpose of the {after} template parameter used by all the existing statement translators (except the terminating statements break, continue, raise, and return).
Have you read PEP 343? It gives a slightly different translation of the with statement. The most important difference is the finally: block, which ensures that __exit__ is called even if the body jumps out of the with: block using break, continue, or return. A more subtle difference is when the __exit__ method is looked up—this matters because __getattr__/__getattribute__ could have side effects, and we’ve tried to be careful about preserving evaluation order even to this level of detail (tests/eval_order.py).
In general, I think it would be better to factor out the try…except translation code into functions that can be reused without literally generating and re-parsing fake ast nodes. You’ve already discovered some of the reasons. For example, your var and store_var kludge will make bugs elsewhere harder to find—those KeyErrors are really important for making sure we stay meticulously synchronized with the symtable module by visiting child namespaces in the same order (tests/symtable_torture.py).
Such refactoring could also tremendously simplify the resulting translated code. For example, the sys.exc_info() call could be eliminated because the __exit__ handler used for catching exceptions already gets passed this information.
@andersk thanks for the thorough read-through. Responses to notes in line:
Have you read PEP 343? It gives a slightly different translation of the with statement. The most important difference is the finally: block, which ensures that exit is called even if the body jumps out of the with: block using break, continue, or return.
Ah, that's a good point. I hadn't considered jumping out of the with block from a loop.
A more subtle difference is when the exit method is looked up—this matters because getattr/getattribute could have side effects, and we’ve tried to be careful about preserving evaluation order even to this level of detail
Makes sense as well.
In general, I think it would be better to factor out the try…except translation code into functions that can be reused without literally generating and re-parsing fake ast nodes.
I'm not sure what you mean here. Are you just suggesting that we replace the big nested AST literals with easier-to-read/change functions? Or do you think the strategy of doing AST-level writing itself is the wrong way to do this. My experience has been that it's significantly easier to do source-to-source transformations on AST trees than on raw strings
I’m explaining our reasons to avoid AST-level rewriting, yes. Perhaps things would be different if symtable worked on AST input—though even without such external considerations, I find it better to refactor an AST-to-AST translation
translate(ast.Foo(args)) = translate(ast.Bar(f(args)))
translate(ast.Bar(args)) = some output
as something more like
translate(ast.Foo(args)) = translateBar(f(args))
translate(ast.Bar(args)) = translateBar(args)
translateBar(args) = some output
Some examples of this pattern already in use are assignment_component and comprehension_code. (Also, we aren’t working on raw strings, we’re working on the T class.)