Extend connection information to support tvn binary and partitioned subgraph

[chunseoklee]

parent : #9123

circle-partition generates the connectivity information in ~~ini~~ json format like the following(from https://github.com/Samsung/ONE/pull/6231#issuecomment-800762497)

      {
         "source" : {
           "file" : "Net_Sqrt_Rsqrt_001.circle",
           "inputs" : [ "ifm" ],
           "outputs" : [ "sqrt", "rsqrt", "ofm" ]
         },
         "parts" : [
           {
             "file" : "Net_Sqrt_Rsqrt_001.part.00001_trix.circle",
             "inputs" : [ "ifm" ],
             "outputs" : [ "sqrt2", "sqrt" ],
           },
           {
             "file" : "Net_Sqrt_Rsqrt_001.part.00002_cpu.circle",
             "inputs" : [ "sqrt2" ],
             "outputs" : [ "ofm", "rsqrt"]
           }
         ]
       }

It uses node's string name as key.

1. Since tvn binary does not provide string node name, we extend filed to support it.
2. If non 0-th(main) subgraph is partitioned, we need to find a way to support this. I am not sure that node name is unique among the whole subgraphes

[seanshpark]

in ini format like

This is json format. There's another ini format.

---

Since tvn binary does not provide string node name, we extend filed to support it.

Reading the circle I/O, we can get name vs tensor index.

Or would it be better to save something like this? (I need to find out if this is possible)

"inputs" : [ "0:ifm" ],

which means, tensor index is 0, name is ifm

---

I am not sure that node name is unique among the whole subgraphes

I don't think it's unique between sub graphs. We can think of like this format

"1:10:sqrt"

which means 1'th sub graph, tensor index 10, name is sqrt

---

If non 0-th(main) subgraph is partitioned, we need to find a way to support this.

circle-partitioner does not support partitioning other sub graphs than 0 (the main). I'm not sure I could develop to support this in this year cause of my schedule.

[chunseoklee]

"1:10:sqrt"

I also have this kind of expression in mind : (subgraph id, tensor idx, strong name) triple.

But, it is not clear how to represent the connectivity between this triples when we deal with multiple subgraphes or tvn.

We need to provide additional connectivity list like :

(tvn's 1st output -> A.circle's 2nd input)

[chunseoklee]

circle-partitioner does not support partitioning other sub graphs than 0 (the main). I'm not sure I could develop to support this in this year cause of my schedule.

If nnpackage spec is defined, I will do verification with manually partitioned models.

[glistening]

At a glance, I thought {model_idx}:{subgraph_idx}:{input_or_output_idx} may be enough. ( Name could be used for debug and human. ) ( I am not sure when it involves loop. ) ( We may introduce some requirement to make runtime simple. For example, list the models in execution order. )

But the comments above seemingly use tensor idx or/and name.

Please let me know if I am missing something.

From nnpkg's view, nnpackage can have several models.

`models` is an array of path to model files

https://github.com/Samsung/ONE/blob/master/nnpackage/spec/10_packaging_and_manifest.md#models

Thus, we can refer to a model by an index.

Then, each models (especially circle and tflite) can have several SubGraphs.

  // All the subgraphs of the model. The 0th is assumed to be the main
  // model.
  subgraphs:[SubGraph];

Again, we can refer to a subgraph by an index.

Then we can refer to inputs and outputs by index, too.

table SubGraph {
  // A list of all tensors used in this subgraph.
  tensors:[Tensor];

  // Indices of the tensors that are inputs into this subgraph. Note this is
  // the list of non-static tensors that feed into the subgraph for inference.
  input:[int];

  // Indices of the tensors that are outputs out of this subgraph. Note this is
  // the list of output tensors that are considered the product of the
  // subgraph's inference.
  outputs:[int];

Again, it is an index.

[hseok-oh]

Before handling subgraph partitioning, we should consider that tflite assume that model input and output are belong to main subgraph (subgraph 0).

If we divide model file on main graph, each new partitioned model file becomes new tflite (or circle) model file.

But if we divide model on non-main subgraph, new partitioned models may breaks this assumption. Partitioned subgraph model file can be a little bit different type model file.

[glistening]

@hseok-oh Let me think nnpkg spec and partitioner separately.

I am thinking in view of nnpkg. For nnpkg, each models are already standalone (self-complete) models . To give connection information, we don't need to worry that whether each model comes from one model, or comes from sub model, ....

[seanshpark]

But, it is not clear how to represent the connectivity between this triples when we deal with multiple subgraphes or tvn.

our tvn currently supports only single subgraph model.

need to check onecc or one-build is able to do this

• circle -> tvn -> pack
• circle -> partition -> tvn
• circle -> partition -> pack ; I think this is currently under development)
• circle -> partition -> tvn -> pack

[glistening]

I have a different idea.

For example, https://github.com/Samsung/ONE/issues/9126#issuecomment-1128336729 and so on.

At first, I imagined a tool which partitions and generates nnpkg as its output w/o pack.

But I found that

circle-partition generates the connectivity information in ini json format like the following(from https://github.com/Samsung/ONE/pull/6231#issuecomment-800762497)

this PR. It was created on Mar 8, 2021, and has several comments and reviews.

Then, I do not oppose the already on-going design. : )

Also, I think there may be several ways to generate partitioned nnpackages.

[chunseoklee]

Here is a proposal for connection info spec :

• Suppose that Net_Sqrt_Rsqrt_001.part.00001_trix.circle is compiled into tvn named 'Net_Sqrt_Rsqrt_001.part.00001_trix.tvn'
• Proposal
  1. Tensor : "model idx:subgraph idx:I/O idx" idx triple separated by colon
  2. Connection : A list of key values pairs : input tensor as a key and output tensor as a value
• "parts" field is not mandatory if model index is sufficient to represent models

       {
         "source" : {
           "inputs" : [ "0:0:0" ],
           "outputs" : [ "0:0:1", "1:0:1", "1:0:0" ]
         },
         "parts" : [
           {
             "file" : "Net_Sqrt_Rsqrt_001.part.00001_trix.tvn",
             "inputs" : [ "ifm" ],
             "outputs" : [ "sqrt2", "sqrt" ],
           },
           {
             "file" : "Net_Sqrt_Rsqrt_001.part.00002_cpu.circle",
             "inputs" : [ "sqrt2" ],
             "outputs" : [ "ofm", "rsqrt"]
           }
         ],
         "conn" : [
             "1:0:0" : "0:0:1",
         ]
       }

[glistening]

Let me write down what I am thinking in addition to https://github.com/Samsung/ONE/issues/9126#issuecomment-1128336729.

Requirement or Role of circle-partitioner and nnpkg specification

• circle-partitioner is a tool to partition a circle file. (It does not know or handle about other model format)
• nnpackage should be able to represent connection between models which may be either circle / tflite / tvn.

Edge, Node Notation

• nnpkg specification will represent connection by edge, which is defined by two nodes, where
• Each node will have the form of model_id:subgraph_id:input_or_output_index.
• model_id is given by the sequence in nnpackage spec's models list.
• subgraph_id
  • tflite and circle : defined from its schema.
  • tvn, it will always 0 since there is only one graph at this moment.
• input_or_output_idx is corresponds to
  • circle or tflite : index to inputs and outputs as I mentioned in https://github.com/Samsung/ONE/issues/9126#issuecomment-1128336729
  • tvn : it will be the index to input or output segments

Terms

• Suppose the original graph is G, and G is partitioned into P1, P2, ... Pn.
  • Let's say the connection between Partitions intra-edge (Please suggest better name if you have : ) )
• Let's extend the scenario, we have multiple original graphs (G1, G2, ... Gn).
  • Let's say the connection between Graphs inter-edge.

More

• For every inter-edge e1 = (n1, n2), the composing nodes must be intput/output from original graphs
• It is not required for intra-edge
• For human or debug, we may use string like current circle-partitioner, which is easier to read.
• I prefer location to name to represent inter-edge.
  • It may be more related to circle-partitioner.
  • It seems to support opcode or opname only https://github.com/Samsung/ONE/blob/master/compiler/circle-partitioner/README.md#partition-section
  • I would like to try several partitions to find out the partition that shows better performance. We need more freedom to get partition. opname may be the one we can use, but I prefer location (number).
  • It is the same way we've used from tflite select_operator.py. (https://github.com/Samsung/ONE/tree/master/tools/tflitefile_tool#example-1)

Question

• Q1. Do we need to create circle for cpu and acl_cl partition?
  • At least, before tvn, we don't need to create multiple circles from a single circle for multiple backends.
  • I assumed we don't need. Is there reason to create circle and represent intra-edges for non-npu backends ( cpu, acl_cl, ... ) ?
• Q2. Do we need to consider multiple original graphs ?

• Let's extend the scenario, we have multiple original graphs (G1, G2, ... Gn).
  • Let's say the connection between Graphs inter-edge.

• In other words, is it possible to assume that all models in nnpkg come from a single model file. (by partitioning and by compiling compile circle to tvn).
• Q3. How can the user identify nnpkg input s and outputs only with connection information?

[chunseoklee]

Q2. Do we need to consider multiple original graphs ?

Not for now. My proposal assumes a single original model. For multiple original graph, need to extend it

Q3. How can the user identify nnpkg input s and outputs only with connection information?

https://github.com/Samsung/ONE/issues/9126#issuecomment-1134183889 is self-contained since it assumes single original model.

[glistening]

Q2. Do we need to consider multiple original graphs ?

Not for now. My proposal assumes a single original model. For multiple original graph, need to extend it

Under this assumption, it becomes simpler than I thought. :relaxed: I can remote all "inter-...". : )

Now, I can understand more about your suggestion under this assumption. Then,

     "conn" : [
        "1:0:0" : "0:0:1",
    ]

In "conn", what does "1:0:0" : "0:0:1", mean?

[chunseoklee]

In "conn", what does "1:0:0" : "0:0:1", mean?

"1:0:0" (2nd model's first input as key) comes from "0:0:1" (1st model's second output as value).

[glistening]

@hseok-oh, @chunseoklee What do you think about Q1 above in https://github.com/Samsung/ONE/issues/9126#issuecomment-1134206533?

[chunseoklee]

Here is a new updated connection info file :

       {
         "source" : {
           "inputs" : [ "0:0:0" ],
           "outputs" : [ "0:0:1", "1:0:1", "1:0:0" ]
         },
         "conn" : [
             { from: "0:0:1", to: [ "1:0:0" ] } ,
         ]
       }

1. "conn" filed has been changed. Note that value of "to" key is a list.
2. "part" field removed.