dist: PLY is no longer usable with optimized frozen code

[el-dee]

trafficstars

Description

Since commit 4c7e513dfdbb4cc4b11567092487da4dd385ceed code frozen code generated by FreezeTool.py is always compiled using optimize level 2. This make any code using PLY (http://www.dabeaz.com/ply/) unusable as by default PLY uses the docstring of the functions to build the lexer and the parser.

If an app is directly using PLY, the workaround is to use the TOKEN decorator provided by PLY instead of the docstring for the lexer, and manually create a decorator to set __doc__ for the rule of the parser functions (see below), but some third-party libraries are not using this and so will be unusable.

Environment

• Operating system: N/A
• System architecture: N/A
• Panda3D version: 1.11
• Installation method: built from source
• Python version (if using Python): N/A
• Compiler (if using C++): N/A

Decorator code

def Rule(r):
    def set_rule(f):
        f.__doc__ = r
        return f
    return set_rule

[el-dee]

Edited the description as PLY does not provide a decorator to set up the rule of the parser functions, so extra code is needed to work around the limitation.

[Moguri]

I'm guessing the solution to this would be to have a list of modules to not optimize? How would this interact with Py_OptimizeFlag, which is program wide and not per-module? Should module optimization be all or nothing? If so, we could add a new flag to the blobinfo to appropriately set Py_OptimizeFlag.

[rdb]

Note that optimize level 1 doesn't strip the docstrings, so we don't have to disable it altogether.

Can we get away with a module blacklist in this case, to not strip the docstrings?

[Moguri]

Do we know if we gain much between level 1 and 2? In other words, is there much benefit to removing doc strings, or do we just switch over to level 1 for everything instead of maintaining a blacklist?

[el-dee]

Mixing compiled bytecode with different optimization levels is not supported by CPython and should be avoided, see https://github.com/numpy/numpy/issues/13248#issuecomment-713166650 and https://github.com/numpy/numpy/issues/13248#issuecomment-713830983

[rdb]

Is this fixed by #1359, or does fixing this require a new optimization level setting?

[el-dee]

Is this fixed by #1359, or does fixing this require a new optimization level setting?

No, PLY requires the presence of the docstrings (or the usage of the above mentioned workaround), so #1359 is needed to fully solve this issue (and set the optimisation level to 1)

[rdb]

I just discovered Panda isn't actually optimizing code that doesn't have a dot in its module name. :grimacing: Oops.

Will fix that, and add a strip_docstrings setting that can be set to False to lower the optimization level to 1.