libspn
libspn copied to clipboard
pronobis
Duplicating an SPN
TL;DR: how should I duplicate an SPN? I have a big IVs and I want the duplicated SPNs to take different subsets of indicators from this big IVs. How should I do that?
I am now on commit a649c62 (09/10/2018) "batch noise now shuffles instead of rolls".
In a prior version of the libspn that I used (on the master branch), I was able to duplicate an SPN by using my own compute_graph_up function, pasted below. The key difference is that this function deals with the case where there are indicator variables specified in the format of a tuple (node, indices).
from collections import deque, defaultdict
def mod_compute_graph_up(root, val_fun, **kwargs):
all_values = {}
stack = deque() # Stack of inputs to process
stack.append((root, None)) # node and index
while stack:
next_input = stack[-1]
# Was this node already processed?
# This might happen if the node is referenced by several parents
if next_input not in all_values:
if next_input[0].is_op:
# OpNode
input_vals = [] # inputs to the node of 'next_input'
all_input_vals = True
# Gather input values for non-const val fun
for inpt in next_input[0].inputs:
if inpt: # Input is not empty
try:
# Check if input_node in all_vals
if inpt.indices is None:
input_vals.append(all_values[(inpt.node, None)])
else:
input_vals.append(all_values[(inpt.node, tuple(inpt.indices))])
except KeyError:
all_input_vals = False
if inpt.indices is None:
stack.append((inpt.node, None))
else:
stack.append((inpt.node, tuple(inpt.indices)))
else:
# This input was empty, use None as value
input_vals.append(None)
# Got all inputs?
if all_input_vals:
last_val = val_fun(next_input, *input_vals, **kwargs)
all_values[next_input] = last_val
stack.pop()
else:
# VarNode, ParamNode
last_val = val_fun(next_input, **kwargs)
all_values[next_input] = last_val
stack.pop()
else:
stack.pop()
return last_val
Then I wrote a function that uses mod_compute_graph_up which actually copies the entire structure and parameters of a given SPN, while keeping the same inputs. This worked well before, and I have documented how it works. But now after I have switched to newer code (a649c62), my SPN duplication code does not work any more. I got an error:
TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'
I think this is due to some changes to the SPN structure internally. Then, how should I duplicate an SPN? I have a big IVs and I want the duplicated SPNs to take different subsets of indicators from this big IVs. How should I do that?
cc: @jostosh @pronobis
(apologies for keep posting issues)
Could you show me the traceback of the error? Where in the code is it happening?
Also, I was wondering why you want to duplicate SPNs? Do they take in different IVs? In that case you could do:
- Build first SPN and build TF graph
- Replace only the bottom IVs and rebuild TF graph
- Repeat 2 for as long as necessary
I will post the traceback later, but for me right now, I would like to use a single IVs object and have different SPNs with the same structure and parameters take different subsets of indices from the IVs.
I am not sure how you to do step 2 as you described. Right now I construct the SPN structure via dense generation and calling dense generator again might result in a different structure. Therefore I need to copy the structure of a generated SPN.
Another reason I was duplicating SPNs is because I need to infer the value of some indicators in the IVs given values of other indicators in this IVs. I thought it would be more straightforward to implement the inference if the duplicated SPNs form together a big SPN graph. This may not be the most optimal way to implement this but it is what my code is based on for now.
Here is the traceback:
Partition 1
Will duplicate ThreeNodeTemplate 26 times.
Duplicating ThreeNodeTemplate... 1 Traceback (most recent call last):
File "/home/zkytony/Documents/thesis/experiments/deep-semantic-mapping/deepsm/experiments/train_test_graphspn.py", line 33, in <module>
main()
File "/home/zkytony/Documents/thesis/experiments/deep-semantic-mapping/deepsm/experiments/train_test_graphspn.py", line 29, in main
available_commands[args.command]()
File "/home/zkytony/Documents/thesis/experiments/deep-semantic-mapping/deepsm/experiments/cold_database_experiment.py", line 660, in same_building
seq_id=args.seq_id, skip_placeholders=args.skip_placeholders)
File "/home/zkytony/Documents/thesis/experiments/deep-semantic-mapping/deepsm/experiments/cold_database_experiment.py", line 481, in run_experiment
partitions=pset)
File "/home/zkytony/Documents/thesis/experiments/deep-semantic-mapping/deepsm/graphspn/tbm/spn_instance.py", line 232, in __init__
db_name=kwargs.get('db_name', None), extra_partition_multiplyer=kwargs.get('extra_partition_multiplyer', 1))
File "/home/zkytony/Documents/thesis/experiments/deep-semantic-mapping/deepsm/graphspn/tbm/spn_instance.py", line 354, in _init_struct
template_spn_roots.extend(NodeTemplateInstanceSpn._duplicate_template_spns(self, tspns, template, supergraph, nodes_covered))
File "/home/zkytony/Documents/thesis/experiments/deep-semantic-mapping/deepsm/graphspn/tbm/spn_instance.py", line 426, in _duplicate_template_spns
labels=[labels])
File "/home/zkytony/Documents/thesis/experiments/deep-semantic-mapping/deepsm/graphspn/spn_model.py", line 395, in mod_compute_graph_up
last_val = val_fun(next_input, *input_vals, **kwargs)
File "/home/zkytony/Documents/thesis/experiments/deep-semantic-mapping/deepsm/graphspn/tbm/spn_template.py", line 278, in dup_fun_up
return spn.Sum(*args[2:], weights=args[0])
File "/home/zkytony/Documents/thesis/experiments/libspn/libspn/graph/sum.py", line 42, in __init__
name=name)
File "/home/zkytony/Documents/thesis/experiments/libspn/libspn/graph/basesum.py", line 63, in __init__
self._reset_sum_sizes(num_sums=num_sums, sum_sizes=sum_sizes)
File "/home/zkytony/Documents/thesis/experiments/libspn/libspn/graph/basesum.py", line 187, in _reset_sum_sizes
self._sum_sizes = sum_sizes or self._get_sum_sizes(self._num_sums)
File "/home/zkytony/Documents/thesis/experiments/libspn/libspn/graph/basesum.py", line 89, in _get_sum_sizes
num_values = sum(input_sizes[2:]) # Skip ivs, weights
TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'
So, you need this feature to work on feature/convspn specifically? I will merge jostosh/libspn/dev-refactor-for-release to it soon which hopefully fixes this issue.
I see. Sounds like you know why that happens? I can also merge this branch to my local branch myself, if that creates less potential trouble for you.