koala icon indicating copy to clipboard operation
koala copied to clipboard

Explore the idea of pure functional evaluation

Open bleucitron opened this issue 8 years ago • 2 comments

This is to avoid to eval() at each node of the graph, which takes quite a long time.

bleucitron avatar Jul 11 '16 10:07 bleucitron

Two ideas were tested:

  • create a single formula string. This idea works on small excel files, but consumes too much memory for large files. This might not be relevant anymore.
  • pure functional evaluation. Most of the code has been modified, but it doesn't work yet. Some refactoring might be necessary => propagate the function results from sp.set_value(input, value) rather than call the uppermost function (with sp.evaluate(output))

bleucitron avatar Jul 11 '16 16:07 bleucitron

After some testings, it is not proven that this idea will reduce our eval timings:

  • on a small file, eval were slightly higher
  • what we might save not using eval, we will probably waste loading the functions with marshal util

But most of all, it does not seem possible to load our function from gzip, since they are not pure functions...

bleucitron avatar Aug 03 '16 09:08 bleucitron