aesara icon indicating copy to clipboard operation
aesara copied to clipboard

Missing `c_code_cache_version` in `grad(sum(a), a)` graph

Open mattearllongshot opened this issue 3 years ago • 2 comments

Description of your problem or feature request

With this graph we do a compilation for each process that runs (storing it for the duration of the process), rather than fully utilizing the cache. This can take a while if launching many processes.

Write the following to bug.py:

import aesara
import aesara.tensor as at

a = at.vector()
aesara.function([a], [at.grad(at.sum(a), a)])

import time
print('sleeping')
time.sleep(1000)

Run the following:

$ rm -r ~/.aesara/compiledir_Linux-4.15--generic-x86_64-with-glibc2.10-x86_64-3.8.13-64/
$ AESARA_FLAGS="cmodule__warn_no_version=True" python bug.py
WARNING (aesara.link.c.cmodule): not all the following op(s) implement c_code_cache_version(). This makes them recompiled for each process.[<aesara.tensor.basic.Alloc object at 0x7efdc902fe20>]
sleeping
^Z
[2]+  Stopped                 AESARA_FLAGS="cmodule__warn_no_version=True" python bug.py
$ ls ~/.aesara/compiledir_Linux-4.15--generic-x86_64-with-glibc2.10-x86_64-3.8.13-64/tmp*
/home/matthew/.aesara/compiledir_Linux-4.15--generic-x86_64-with-glibc2.10-x86_64-3.8.13-64/tmpagqncksm:
__init__.py  key.pkl  mba10987274f369529454d4a60996746c3927712a0b1b3928446a3c275151e2ee.so  mod.cpp  __pycache__

/home/matthew/.aesara/compiledir_Linux-4.15--generic-x86_64-with-glibc2.10-x86_64-3.8.13-64/tmpb0w3p1wz:
__init__.py  key.pkl  m6c1526e289fbc101e04dd872f32684efbca30d864c16c82ff681a372c78477f7.so  mod.cpp  __pycache__

/home/matthew/.aesara/compiledir_Linux-4.15--generic-x86_64-with-glibc2.10-x86_64-3.8.13-64/tmpimzptk_5:
__init__.py  mb6a3fac88303d6c4edebfbc631c7a259b4e25534130e9774504486e712d37215.so  mod.cpp  __pycache__
$

The dir with mb6a3fac88303d6c4edebfbc631c7a259b4e25534130e9774504486e712d37215.so in it is missing a key.pkl file, and is recompiled for each new proc that runs. The warning message suggests that it is a case of adding a missing c_code_cache_version method for part of the graph. Is this possible?

Versions and main components

  • Aesara version:2.7.9
  • Aesara config (python -c "import aesara; print(aesara.config)") aesara_config.txt
  • Python version: Python 3.8.13
  • Operating system: Ubuntu 18.04.6 LTS
  • How did you install Aesara: conda

mattearllongshot avatar Aug 06 '22 09:08 mattearllongshot

That warning is also reproducible with aesara-cache clear and the following:

import aesara
import aesara.tensor as at

a = at.vector()

with aesara.config.change_flags(cmodule__warn_no_version=True):
    aesara.function([a], [at.grad(at.sum(a), a)])

brandonwillard avatar Aug 09 '22 16:08 brandonwillard

This appears to be happening because the condition here is being met due to this step. In other words, Apply.__props__ is empty for Alloc (i.e. it has no identifying instance properties to hash), and that is being (mistakenly) interpreted as an un-version-able Apply node signature.

brandonwillard avatar Aug 09 '22 17:08 brandonwillard