Make it work with the standard library

[tristanlatr]

I believe very little effort is needed to make it work on python3 with the ast module. Of course, this would only work on python3 for python3 ast. But compatibility with python2 would still be possible by installing gast.

What do you think ?

[tristanlatr]

Hi @serge-sans-paille, after our conversation in #46, I thought about it again and I believe we can make it work with even less changes. By maintaining a mapping from gast nodes to the ast nodes (some nodes might not have a correspondance tho, like the store names inside the except handlers), we can then build an alternative def use chains that uses the standard nodes. The only change we need to make is to provide a function to produce that mapping. Tell me what you think.

[tristanlatr]

This should do the trick: https://github.com/serge-sans-paille/gast/pull/76

[serge-sans-paille]

I understand your need, but I don't see myself maintaining this extra approach. I think it could be possible to achieve a similar approach by subclassing gast.Ast3ToGAst and defining your own ast_to_gast, and this could be done on the client side.

[tristanlatr]

Fair enough, I’ll publish here the code to make it work with stdlib in case it might be useful for others.

Talk to you later,

[tristanlatr]

Here's the code, hope it helps.

EDIT: It doesn't work at 100%.

import ast
import gast
from gast.ast3 import Ast3ToGAst
from beniget.beniget import DefUseChains, Def

class _AstToGAst(Ast3ToGAst):
    def __init__(self) -> None:
        self.mapping = {}
    def visit(self, node):
        newnode = super().visit(node)
        if not isinstance(node, ast.expr_context):
            self.mapping[newnode] = node
        return newnode

def ast_to_gast(node:ast.Module) -> Tuple[gast.Module, Mapping[gast.AST, ast.AST]]:
    """
    This function returns a tuple which first element is the ``gast`` module and the second element is a 
    mapping from gast nodes to standard library nodes. It should be used with caution
    since not all nodes have a corespondance. Namely, ``expr_context`` nodes and the store ``Name`` of 
    ``ExceptHandler``s are not present in the mapping.
    """
    # returns a tuple: (gast node, mapping from gast node to ast node)
    _vis = _AstToGAst()
    newnode = _vis.visit(node)
    return newnode, _vis.mapping

def _convert_beniget_def(definition:Def, 
                         converted:Dict[Def, 'Def|None'], 
                         mapping:Mapping[gast.AST, ast.AST]) -> 'Def|None':
    if definition in converted:
        return converted[definition]
    new_definition = Def(mapping.get(definition.node))
    converted[definition] = new_definition
    if new_definition:
        for user in definition.users():
            new_definition.add_user(_convert_beniget_def(user, converted, mapping))
    return new_definition

def _def_use_chains_to_stdlib(chains:Dict[gast.AST, Def], 
                              mapping:Mapping[gast.AST, ast.AST]) -> Tuple[Dict[ast.AST, Def],
                                                                           Dict[Def, 'Def|None']]:
    new_chains:Dict[ast.AST, Def] = {}
    converted:Dict[Def, 'Def|None'] = {}
    for definition in chains.values():
        new_def = _convert_beniget_def(definition, converted, mapping)
        if new_def:
            new_chains[new_def.node] = new_def
    return new_chains, converted

def DefUseChainsAndLocals(node:ast.Module,
                          filename:Any=None,) -> Tuple[Dict[ast.AST, Def], 
                                                       Dict[ast.AST, Dict[str, List['Def|None']]]]:
    
    gast_node, mapping = ast_to_gast(node)
    # - compute local def-use chains
    defuse = DefUseChains(filename=filename)
    assert hasattr(defuse, 'future_annotations')
    defuse.future_annotations = True
    defuse.visit(gast_node)
    chains, converted = _def_use_chains_to_stdlib(defuse.chains, mapping)
    locals_as_dict: Dict[ast.AST, Dict[str, List['Def|None']]] = {}
    for namespace,loc_list in defuse.locals.items():
        d = locals_as_dict.setdefault(mapping[namespace], {})
        for loc in loc_list:
            d.setdefault(loc.name(), []).append(converted[loc])
    return chains, locals_as_dict

[tristanlatr]

Hi @serge-sans-paille,

Je me permet de rouvrir ce ticket puisque j'ai fait deux trois tests de performances. En roulant libstatic sur le code de twisted avec cette commande: python3 -m libstatic ./.tox/twisted-trunk/src/twisted/ -u twisted.python.deprecate.deprecated --exclude *test* et en analysant la trace avec speedscope, les chiffres montrent que 16% du runtime est consacré a ast_to_gast, 13% a ast.parse(), 7.3% est consacré a convertir les Def-Use chains et les locals avec les noeuds standards, 7.5% est consacré a renverser les Def-Use chains pour faire les Use-Def chains, 5.1% pour générer les Def-Use chains, etc... On pourrait gagner environs 25% de runtime si beniget supportait la librairie standard.

[tristanlatr]

Thé standard library is now supported on the latest branch