WIP: Pre-review for v1.3.0 release

[ankostis]

Continuation of #30, containing review-fixes in huyng/graphkit#1.

• ENH: heavily reinforced exception annotations ("jetsam"):
  • FIX: (8f3ec3a) outer graphs/ops do not override the inner cause.
  • ENH: retrofitted exception-annotations as a single dictionary, to print it in one shot (8f3ec3a & 8d0de1f)
  • enh: more data in a dictionary
  • TCs: Add thorough TCs (8f3ec3a & b8063e5).
• REFACT(net): rename Delete-->Evict, removed Placeholder from nadanodes, privatize node-classes.
• ENH(sideffects): make them always DIFFERENT from regular DATA, to allow to co-exist.
• TCs: pytest now checks sphinx-site builds without any warnings.
• FIX(plan): multithreading was broken due shared plan.executed: moved to new class Solution.
• [ ] graphop in docs
• [ ] typo(test): overriden-->overriDDen
• [ ] DROP sideffects

[codecov-io]

Codecov Report

Merging #31 into master will increase coverage by 12.98%. The diff coverage is 91.36%.

@@             Coverage Diff             @@
##           master      #31       +/-   ##
===========================================
+ Coverage   77.87%   90.86%   +12.98%     
===========================================
  Files           5        6        +1     
  Lines         348      580      +232     
===========================================
+ Hits          271      527      +256     
+ Misses         77       53       -24

Impacted Files	Coverage Δ
graphkit/modifiers.py	100% <100%> (ø)	:arrow_up:
graphkit/__init__.py	100% <100%> (ø)	:arrow_up:
graphkit/base.py	81.57% <83.33%> (+2.21%)	:arrow_up:
graphkit/plot.py	87.78% <87.78%> (ø)
graphkit/network.py	94.69% <94.61%> (+24.48%)	:arrow_up:
graphkit/functional.py	95.55% <96.07%> (+1.8%)	:arrow_up:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update e70718b...7658e17. Read the comment docs.

[ankostis]

@huyng can you merge #30 into master (partially reviewed in huyng/graphkit#1), to trim this long commit-history?

[huyng]

I haven't merged the previous commits because they introduced that multithreading bug.

This PR is growing quite large and is combining several issues.

It would be better to break this up into several PRs:

1. fixes for #25 and #24 while not introducing the multithreading bug
2. additional plotting & debugging niceties as long as it doesn't introduce too much complexity into the code or reduce performance of graph execution.

[ankostis]

The fix for #24 & #25 is #26. Please merge this asap.

[ankostis]

I will try to fix the multithreading bug here, and backport to #26.

[huyng]

PR #26 is still causing the multithreading error from the following code. If I merge the code, it will break our master branch, which I'm trying to avoid here. We'll have to wait until the multithreading issue is resolved. I'd recommend that you work off of your PR#26's branch to resolve the multithreading issue before moving forward with other changes as it will make the review process easier to follow (since the PR deals primarily with the issues #24 and #25).

from multiprocessing.dummy import Pool
from operator import mul, sub
from graphkit import compose, operation

# Computes |a|^p.
def abspow(a, p):
    c = abs(a) ** p
    return c

# Compose the mul, sub, and abspow operations into a computation graph.
graph = compose(name="graph")(
    operation(name="mul1", needs=["a", "b"], provides=["ab"])(mul),
    operation(name="sub1", needs=["a", "ab"], provides=["a_minus_ab"])(sub),
    operation(name="abspow1", needs=["a_minus_ab"], provides=["abs_a_minus_ab_cubed"], params={"p": 3})(abspow)
)

pool = Pool(10)
graph.set_execution_method("parallel")
pool.map(lambda i: graph({'a': 2, 'b': 5}, ["a_minus_ab","abs_a_minus_ab_cubed"]), range(100))

~/Huyng/graphkit/graphkit/network.py in <lambda>(op)
    501
    502             done_iterator = pool.imap_unordered(
--> 503                                 lambda op: (op,op._compute(cache)),
    504                                 upnext)
    505             for op, result in done_iterator:

~/Huyng/graphkit/graphkit/functional.py in _compute(self, named_inputs, outputs)
     17
     18     def _compute(self, named_inputs, outputs=None):
---> 19         inputs = [named_inputs[d] for d in self.needs if not isinstance(d, optional)]
     20
     21         # Find any optional inputs in named_inputs.  Get only the ones that

~/Huyng/graphkit/graphkit/functional.py in <listcomp>(.0)
     17
     18     def _compute(self, named_inputs, outputs=None):
---> 19         inputs = [named_inputs[d] for d in self.needs if not isinstance(d, optional)]
     20
     21         # Find any optional inputs in named_inputs.  Get only the ones that

KeyError: 'a_minus_ab'

[ankostis]

Exactly!

I will work on the tip though, to ensure my changes remain future-proof, and will backport them to 26.

[ankostis]

Just pushed a fix here for the new "multithreading" TC.

And my mistake on #26, it never shared executed variable, so it passes ok with the new "multithreading" TC, you may merge it.

[ankostis]

For reference i continued the implementation of thi PR in pygraphkit/graphtik project, which have departed since, as described in pygraphkit/graphtik#1.