FullyConnected Layer Reordering for TF Lite Exporter

[DonghakPark]

When Export NNTrainer to Tensorflow Lite we need to reordering

--> Implement based on #1892

• [x] Implement Flatten Reordering
• [x] Test various FC Case
• [x] Make Unit Test

[taos-ci]

:octocat: cibot: Thank you for posting issue #1973. The person in charge will reply soon.

[DonghakPark]

In /nntrainer/nntrainer/utils/node_exporter.cpp below function

void Exporter::saveTflResult(const std::tuple<props::Unit> &props,
                             const FullyConnectedLayer *self) 

before save FullyConnectedLayer change "NCHW < -- > NHWC" not just transpose but reordering for Fully Connected layer

image from (https://oneapi-src.github.io/oneDNN/dev_guide_understanding_memory_formats.html)

[DonghakPark]

in case of this data [[[ 0 1] [ 2 3] [ 4 5] [ 6 7]]

[[ 8 9] [10 11] [12 13] [14 15]]

[[16 17] [18 19] [20 21] [22 23]]]

in NNtrainer the data order look like below [ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 ] and Tensorflow case it look like below [ 0 8 16 1 9 17 2 10 18 3 11 19 4 12 20 5 13 21 6 14 22 7 15 23]

so i will calculate position and relocate weight matrix

For change Matrix I should know previous input shape

    for (int h = 0; h < HEIGHT; h++) {
      for (int w = 0; w < WIDTH; w++) {
        for (int c = 0; c < CHANNEL; c++) {

          int now_position = c * (HEIGHT * WIDTH) + h * WIDTH + w;
          int next_position = h * (WIDTH * CHANNEL) + w * CHANNEL + c;
          
------> switching data now to next
        }
      }
    }

[DonghakPark]

now Layer Name is : input Input shape : 1:3:2:4 output shape : 1:3:2:4 now Layer Name is : flatten Input shape : 1:3:2:4 output shape : 1:1:1:24 now Layer Name is : fully_connected Input shape : 1:1:1:24 output shape : 1:1:1:2 CHW == 3:2:4

BEFORE : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

AFTER : 0 8 16 1 9 17 2 10 18 3 11 19 4 12 20 5 13 21 6 14 22 7 15 23 24 32 40 25 33 41 26 34 42 27 35 43 28 36 44 29 37 45 30 38 46 31 39 47

[DonghakPark]

@jijoongmoon Can you check the contents below?

I think if we convert NNtrainer to tflite it should be change input_shape But now if we made NNtrainer model with 1:3:2:4 (NCHW) and export to tflite model and load tflite model then it's input shape still 1:3:2:4(NHWC)

Is it the intended part or am I misunderstood?

NNtrainer input (1:3:2:4[NCHW]) --> conver to TF Lite (weight converted) --> TF Lite input shape in Python Code (1:3:2:4[NHWC])