[draft] Add Weight Visualization

[staddy]

This pull request adds a weight visualization implementation example: On expanding a compatible node's tensor (with dim >= 2) a weight visualization GUI appears in property side-bar. To visualize a tensor as a 2d image you need to:

1. Select axes of the tensor that will be used as x and y of the image (use the checkboxes of "axes selector", exactly two axes should be selected, the first selected axis is (x) and the second is (y)).
2. Select the values of the other axes that will be fixed with "values selector" - all not selected axes should have a fixed value.
3. The weights will appear as a 2d heatmap below. Blue color is the minimum value, red color is the maximum value. For example: you are visualizing a tensor of shape (32, 3, 3, 3). Let's assume you have selected the 2 lower checkboxes and fixed first axis value as 10 and the second axis value as 1. So the top left pixel of the image corresponds to tensor[10][1][0][0] value. The bottom right pixel - tensor[10][1][2][2] value.

[seanshpark]

@staddy , is this your current change? why I'm asking

• I expect property window show new properties (I didn't pull and checked)
• when clicked, there will be no feedback in the UI; ~I don't expect this to be a full working draft~ I don't see related codes as of now in the change

CircleGraph is shared code, as of now, used in circle graph view, partition editor and visquv

• I don't expect this property be visible in partition editor nor in visquv

What is your user scenario ?

• please describe in detail
• from opening the folder and ....

[staddy]

@seanshpark, yes, this is what is implemented in the draft. Property window shows additional property: VisualTensorView. The implemented UI works as described, the draft is kind of full working if we are talking about visualizing any 2d subarray of a tensor. Currently the property appears for any suitable tensor, so if the code is shared, I need to implement disabling the feature. Current scenario:

• open a folder with a *.circle file
• click on the *.circle file
• click on a node that contains 2+d tensors
• expand the 2+d tensor properties in property window
• the implemented UI appears and functions

[staddy]

• I don't see related codes as of now in the change

Currently the code just changes the default behavior of sidebar, adding a new property to any tensor with >= 2 axes. I thought this implementation is more natural. Now I can add a setting to disable the visualizer by default. Should it be a global setting? Or, maybe, a control that allow to switch between text and visual representation? Or just always enable the visualizer when initializing the sidebar in graph view?

[seanshpark]

Should it be a global setting? Or, ...

@staddy , have you read my comment https://github.com/Samsung/ONE-vscode/pull/1492#issuecomment-1421567999 ? or you don't understand it? please ask if you can't catch the meaning

CircleGraph is shared code, as of now, used in circle graph view, partition editor and visquv

• I don't expect this property be visible in partition editor nor in visquv

[seanshpark]

I would like to see this draft provide your full Current scenario:

[staddy]

Now VisualTensorView property is enabled only in circle graph and disabled in partition editor and visquv. The draft provides full "Current scenario".

[staddy]

Current state: Added UI to modify scale and swap axes.

[seanshpark]

@staddy , I've got your change but I can't see the weight bars. Please describe the condition of the model that shows the bar.

[staddy]

@seanshpark, 'weights' a property of some nodes (in this case it's DepthwiseConv2D). Currently the visualizer works for any properties that are tensors of shape [a, b...] (in this case [1, 3, 3, 32]): So in order to see the visualizer:

• we need a *.circle model that contains at least one node with tensor properties like 'weights', 'filter', etc. (Conv2D, for example).
• open folder that contains such model
• click on the model file in Explorer
• click on such node (Conv2D)
• in 'Node Properties' panel find 'Inputs' containing the tensor property ('filter' in case of Conv2D)
• expand it with + button
• the visualizer should appear

[seanshpark]

Now I can see the bars :)

I've tried with some models and found this:

Model: inception_v3_2018_04_27.tgz

Image is black with this Conv2D

when selected with 0, 3

and when I change 3'rd value black bar disappears

[seanshpark]

Q) How does the bar graph show when Tensor is 2D, 3D or 5D ?

[seanshpark]

Q) what is the meaning of the colors ?

[staddy]

@seanshpark, thank you, the bug with black bar is fixed.

Q) How does the bar graph show when Tensor is 2D, 3D or 5D ?

If I understand your question right,

• when Tensor is 2D, axes selector and values selector are not shown, all values of the Tensor are rendered as a 2D image
• when Tensor is 3D, 4D, 5D ..., axes selector and values selector are shown and user has to select 2 axes with check boxes. The selected axes are used to iterate over (as if Tensor was 2D). For all other axes user has to select fixed values. For example, we have a 5D Tensor. User has selected 0 and 2 axes (with checkboxes), and selected fixed values 5, 13, 7 for the other 3 axes. So the values used for bar will be tensor[x][5][y][13][7], where x - all integers from 0 to length of axis 0, y - all integers from 0 to length of axis 2. So the resulting image pixel (17, 19) is the value tensor[17][5][19][13][7].

[staddy]

Q) what is the meaning of the colors ?

The higher wavelength, the higher value:

• red = 1.0 (the highest value in tensor)
• green = 0.5
• blue = 0.0 (the lowest value in tensor)

'value' is from 0.0 to 1.0 (normalized value from tensor):

      value *= 2;
      let blue = Math.round(Math.max(0, 255 * (1.0 - value)));
      let red = Math.round(Math.max(0, 255 * (value - 1.0)));
      let green = 255 - blue - red;

[seanshpark]

The lower wavelength, the higher value:

thanks for the comment. maybe some document may help understanding for later readers.

[seanshpark]

If I understand your question right,

Yes I think so :)

I've tried with Add operator with input + constant but failed to see the color bars. There seems some problem with attaching the files. so below is how to generate them.

If you can, please try (1) code in Samsung/ONE, res/TensorFlowLiteRecipes/Add_002/test.recipe file (2) Copy to Add_003 and edit it like

operand {
  name: "ifm1"
  type: FLOAT32
  shape { dim: 128 dim: 128 }
}
operand {
  name: "ifm2"
  type: FLOAT32
  shape { dim: 128 dim: 128 }
  filler {
    tag: "gaussian"
    arg: "0.0"
    arg: "1.0"
  }
}
operand {
  name: "ofm"
  type: FLOAT32
  shape { dim: 128 dim: 128 }
}
operation {
  type: "Add"
  input: "ifm1"
  input: "ifm2"
  output: "ofm"
  add_options {
    activation: NONE
  }
}
input: "ifm1"
output: "ofm"

(3) Copy to Add_004 and change shape to

shape { dim: 128 dim: 2 dim: 2 dim: 2 dim: 128 }

(4) build ONE

./nncc configure
./nncc build

• refer https://github.com/Samsung/ONE/blob/master/docs/howto/how-to-build-compiler.md

(5) in build/compiler/common-artifacts folder there should be Add_003.circle and Add_004.circle (6) open the second input, which should be a constant tensor

[staddy]

failed to see the color bars.

@seanshpark, strange,

[seanshpark]

check on first axis 127 and last axis 127

[staddy]

check on first axis 127 and last axis 127

@seanshpark, sorry, still can't reproduce (though currently the width of text input is not enough to show 3 digits - so it's 12 instead of 127 on the screenshots): Have you pulled my last fix (https://github.com/Samsung/ONE-vscode/pull/1492/commits/f6195354dd401eb340774b65aba89bb6bf377b7b)?

[seanshpark]

Have you pulled my last fix

Latest commit is work OK :)

[staddy]

Docs are ready: https://github.com/staddy/ONE-vscode/blob/visualization-draft/docs/VisualTensorView.md

[seanshpark]

Docs are ready:

Thanks for the document! As there are lots of image files, how about move them into new docs/images folder ? Existing images maybe moved later.

Add: I would be better to rename VisualTensorView.jpg to something else.

[staddy]

@seanshpark, done

[seanshpark]

Code looks good to me (not sure for others) Can you please post to 3 PRs? (1) media files (2) src files and (3) doc files.

[staddy]

@seanshpark, done: VisualTensorView implementation (disabled by default): https://github.com/Samsung/ONE-vscode/pull/1502 Enable VisualTensorView for Circle Graph only: https://github.com/Samsung/ONE-vscode/pull/1503 Documentation: https://github.com/Samsung/ONE-vscode/pull/1504