pylint-dev
MemoryError when inferring complex binary operation
Originally reported by: Claudiu Popa (BitBucket: PCManticore, GitHub: @PCManticore)
#!python
from astroid.test_utils import extract_node
n = extract_node('''
def detectencoding_str(input, final=False):
# A bit for every candidate
CANDIDATE_UTF_8_SIG = 1
CANDIDATE_UTF_16_AS_LE = 2
CANDIDATE_UTF_16_AS_BE = 4
CANDIDATE_UTF_16_LE = 8
CANDIDATE_UTF_16_BE = 16
CANDIDATE_UTF_32_AS_LE = 32
CANDIDATE_UTF_32_AS_BE = 64
CANDIDATE_UTF_32_LE = 128
CANDIDATE_UTF_32_BE = 256
CANDIDATE_CHARSET = 512
candidates = 1023 # all candidates
li = len(input)
if li>=1:
# Check first byte
c = input[0]
if c != "\xef":
candidates &= ~CANDIDATE_UTF_8_SIG
if c != "\xff":
candidates &= ~(CANDIDATE_UTF_32_AS_LE|CANDIDATE_UTF_16_AS_LE)
if c != "\xfe":
candidates &= ~CANDIDATE_UTF_16_AS_BE
if c != "@":
candidates &= ~(CANDIDATE_UTF_32_LE|CANDIDATE_UTF_16_LE|CANDIDATE_CHARSET)
if c != 42:
candidates &= ~(CANDIDATE_UTF_32_AS_BE|CANDIDATE_UTF_32_BE|CANDIDATE_UTF_16_BE)
if li>=2:
# Check second byte
c = input[1]
if c != "\xbb":
candidates &= ~CANDIDATE_UTF_8_SIG
if c != "\xfe":
candidates &= ~(CANDIDATE_UTF_16_AS_LE|CANDIDATE_UTF_32_AS_LE)
if c != "\xff":
candidates &= ~CANDIDATE_UTF_16_AS_BE
if c != 42:
candidates &= ~(CANDIDATE_UTF_16_LE|CANDIDATE_UTF_32_AS_BE|CANDIDATE_UTF_32_LE|CANDIDATE_UTF_32_BE)
if c != "@":
candidates &= ~CANDIDATE_UTF_16_BE
if c != "c":
candidates &= ~CANDIDATE_CHARSET
if li>=3:
# Check third byte
c = input[2]
if c != "\xbf":
candidates &= ~CANDIDATE_UTF_8_SIG
if c != "c":
candidates &= ~CANDIDATE_UTF_16_LE
if c != 42:
candidates &= ~(CANDIDATE_UTF_32_AS_LE|CANDIDATE_UTF_32_LE|CANDIDATE_UTF_32_BE)
if c != "\xfe":
candidates &= ~CANDIDATE_UTF_32_AS_BE
if c != "h":
candidates &= ~CANDIDATE_CHARSET
if li>=4:
# Check fourth byte
c = input[3]
if input[2:4] == 42:
candidates &= ~CANDIDATE_UTF_16_AS_LE
if c != 42:
candidates &= ~(CANDIDATE_UTF_16_LE|CANDIDATE_UTF_32_AS_LE|CANDIDATE_UTF_32_LE)
if c != "\xff":
candidates &= ~CANDIDATE_UTF_32_AS_BE
if c != "@":
candidates &= ~CANDIDATE_UTF_32_BE
if c != "a":
candidates &= ~CANDIDATE_CHARSET
candidates #@
''')
print(n.infered())
- Bitbucket: https://bitbucket.org/logilab/astroid/issue/203
its failing for me as well, when running pylint on a file on machine having 1 GB memory free at the time it fails. so it stucks (for 2 minutes) at 1 of the line before failing.
astroid (1.5.2)
pylint (1.7.2)
Traceback (most recent call last):
File "/usr/local/bin/pylint", line 11, in <module>
sys.exit(run_pylint())
File "/usr/local/lib/python2.7/dist-packages/pylint/__init__.py", line 13, in run_pylint
Run(sys.argv[1:])
File "/usr/local/lib/python2.7/dist-packages/pylint/lint.py", line 1300, in __init__
linter.check(args)
File "/usr/local/lib/python2.7/dist-packages/pylint/lint.py", line 726, in check
self._do_check(files_or_modules)
File "/usr/local/lib/python2.7/dist-packages/pylint/lint.py", line 855, in _do_check
self.check_astroid_module(ast_node, walker, rawcheckers, tokencheckers)
File "/usr/local/lib/python2.7/dist-packages/pylint/lint.py", line 934, in check_astroid_module
walker.walk(ast_node)
File "/usr/local/lib/python2.7/dist-packages/pylint/utils.py", line 973, in walk
self.walk(child)
File "/usr/local/lib/python2.7/dist-packages/pylint/utils.py", line 973, in walk
self.walk(child)
File "/usr/local/lib/python2.7/dist-packages/pylint/utils.py", line 973, in walk
self.walk(child)
File "/usr/local/lib/python2.7/dist-packages/pylint/utils.py", line 973, in walk
self.walk(child)
File "/usr/local/lib/python2.7/dist-packages/pylint/utils.py", line 970, in walk
cb(astroid)
File "/usr/local/lib/python2.7/dist-packages/pylint/checkers/stdlib.py", line 184, in visit_call
for inferred in node.func.infer():
File "/usr/local/lib/python2.7/dist-packages/astroid/decorators.py", line 85, in wrapped
res = next(generator)
File "/usr/local/lib/python2.7/dist-packages/astroid/decorators.py", line 138, in raise_if_nothing_inferred
yield next(generator)
File "/usr/local/lib/python2.7/dist-packages/astroid/inference.py", line 237, in infer_attribute
for owner in self.expr.infer(context):
File "/usr/local/lib/python2.7/dist-packages/astroid/context.py", line 40, in cache_generator
for result in generator:
File "/usr/local/lib/python2.7/dist-packages/astroid/decorators.py", line 85, in wrapped
res = next(generator)
File "/usr/local/lib/python2.7/dist-packages/astroid/bases.py", line 95, in _infer_stmts
for inferred in stmt.infer(context=context):
File "/usr/local/lib/python2.7/dist-packages/astroid/context.py", line 40, in cache_generator
for result in generator:
File "/usr/local/lib/python2.7/dist-packages/astroid/decorators.py", line 85, in wrapped
res = next(generator)
File "/usr/local/lib/python2.7/dist-packages/astroid/bases.py", line 95, in _infer_stmts
for inferred in stmt.infer(context=context):
File "/usr/local/lib/python2.7/dist-packages/astroid/context.py", line 40, in cache_generator
for result in generator:
File "/usr/local/lib/python2.7/dist-packages/astroid/decorators.py", line 108, in yes_if_nothing_inferred
for node in func(*args, **kwargs):
File "/usr/local/lib/python2.7/dist-packages/astroid/decorators.py", line 85, in wrapped
res = next(generator)
File "/usr/local/lib/python2.7/dist-packages/astroid/inference.py", line 409, in _filter_operation_errors
for result in infer_callable(self, context):
File "/usr/local/lib/python2.7/dist-packages/astroid/inference.py", line 688, in _infer_binop
context, _get_binop_flow):
File "/usr/local/lib/python2.7/dist-packages/astroid/inference.py", line 628, in _infer_binary_operation
results = list(method())
File "/usr/local/lib/python2.7/dist-packages/astroid/decorators.py", line 108, in yes_if_nothing_inferred
for node in func(*args, **kwargs):
File "/usr/local/lib/python2.7/dist-packages/astroid/protocols.py", line 164, in tl_infer_binary_op
elts = list(_filter_uninferable_nodes(self.elts, context))
File "/usr/local/lib/python2.7/dist-packages/astroid/protocols.py", line 152, in _filter_uninferable_nodes
for inferred in elt.infer(context):
File "/usr/local/lib/python2.7/dist-packages/astroid/context.py", line 44, in cache_generator
self.inferred[key] = tuple(results)
MemoryError
freed up memory on the machine to make it work but is there way to prevent in advance?
@p00j4 Sorry for the late reply, but this problem most likely won't go away by giving to it more memory. :) pylint & astroid currently cannot cope with complex code like the one from your first comment, we need to put some guards to prevent it from crashing or from eating all the memory available.
@PCManticore Any progress so far? I try to use pylint for a little bigger project and it consumes more that 2GB of memory. It would be really nice if one could limit pylint memory consumption. Btw. I really cannot imagine why pylint/astroid needs more than 1GB for the example posted in this issue.
@Mi-La No progress, I have not worked on this in some time. I'm not sure exactly how pylint is supposed to limit its own memory consumption, so that's probably not going to happen. The example from this issue might result in high memory consumption due to the fact that pylint and astroid don't quite have a recursion detection mechanism baked in in the inference engine that pylint uses (e.g. astroid), although that is something that should land at some point.
Feel free to reopen if still an issue in 2.15.x or greater. Thanks.
