ONE Analyze onecc options

related #9313

Let me analyze onecc options and figure out how to handle this for one init.

Jun 30 '22 08:06 hyunsik-yoon

one-optimizer options

Study result:

Format

Optimization pass name
- model type dependency
- backend dependency
- related links

Let's start from https://github.com/Samsung/ONE/blob/e5c70608a0ccbef4a5245767b62d8423eadb60ce/compiler/circle2circle/src/Circle2Circle.cpp#L382

Optimization passes

FoldAddV2, FoldCast, FoldDequantize, FoldDepthwiseConv2D, FoldGather, FoldSparseToDense
- no
- no
ForwardReshapeToUnaryOp
- no. originated from ONNX but seems OK with others
- no
- https://github.com/Samsung/ONE/issues/5848
FuseActivationFunction
- no
- no
- https://github.com/Samsung/ONE/issues/4861
FuseBatchNormWithConv
- no
- no. introduced from NPU Compiler (I wonder this hurt performance in case of ONERT)
- https://github.com/Samsung/ONE/issues/5618
FuseAddWithFullyConnected
- no
- no.
- https://github.com/Samsung/ONE/issues/7724
FuseAddWithTConv
- no
- no
- https://github.com/Samsung/ONE/pull/4375
FuseBatchNormWithDwConv
- no (mainly for ONNX)
- no
- https://github.com/Samsung/ONE/issues/5747
FuseBatchNormWithTConv
- no
- no (for npu)
- https://github.com/Samsung/ONE/issues/3565
FuseBCQ
- no
- no
- https://github.com/Samsung/ONE/issues/1079
FuseInstanceNorm
- no
- no
- https://github.com/Samsung/ONE/issues/1167, https://github.com/Samsung/ONE/issues/1741
FuseMeanWithMean
- no (guess that it is from onnx)
- no
- https://github.com/Samsung/ONE/issues/7335
MakeBatchNormGammaPositive
- no
- onert (for onert, https://github.com/Samsung/ONE/issues/4773 for some language model)
- should not be turned on by default. Not used due to low quantization accuracy. (https://github.com/Samsung/ONE/issues/5784#issuecomment-925431354)
FusePreActivationBatchNorm (same origin with MakeBatchNormGammaPositive)
- no
- onert (for onert, https://github.com/Samsung/ONE/issues/4773 for some language model)
- should not be turned on by default. Not used due to low quantization accuracy. (https://github.com/Samsung/ONE/issues/5784#issuecomment-925431354)
FuseTransposeWithMean (same origin with FuseMeanWithMean)
- no (guess that it is from onnx)
- no
- https://github.com/Samsung/ONE/issues/7335
RemoveFakeQuant
- no
- no but backend specific: https://github.com/Samsung/ONE/issues/5784#issuecomment-925431354
- https://github.com/Samsung/ONE/issues/6801
RemoveQuantDequantSeq (similar with RemoveFakeQuant`)
- no
- no but backend specific: https://github.com/Samsung/ONE/issues/5784#issuecomment-925431354
- https://github.com/Samsung/ONE/pull/6940
RemoveRedundantQuantize
- no
- no
- https://github.com/Samsung/ONE/issues/8526
RemoveRedundantReshape
- no
- no
- https://github.com/Samsung/ONE/issues/5266
RemoveRedundantTranspose
- no
- no
- https://github.com/Samsung/ONE/issues/5062
RemoveUnnecessaryReshape
- no
- no
- https://github.com/Samsung/ONE/issues/5575
RemoveUnnecessarySlice
- no
- no
- https://github.com/Samsung/ONE/issues/5416
RemoveUnnecessaryStridedSlice
- no
- no
- https://github.com/Samsung/ONE/issues/5757
RemoveUnnecessarySplit
- no
- no
- https://github.com/Samsung/ONE/issues/5456
ReplaceMulAddWithDepthwiseConv
- no
- no, designed for npu but works for other backend. if backend's dconv is slower than mul and add, this could make latency slow (LATENCY)
- https://github.com/Samsung/ONE/pull/5244,
  - "This pass is needed to support quantization of ResNetV2 model for some special hardware."
  - pass details
ReplaceSubWithAdd
- no
- no
ResolveCustomOpAdd
- no
- no
- https://github.com/Samsung/ONE/blob/3744b4a4721b7371cdb6ad41b0482ca5fb010362/compiler/luci/pass/src/ResolveCustomOpAddPass.cpp#L46
ResolveCustomOpBatchMatMul
- no
- no
- https://github.com/Samsung/ONE/pull/40#issuecomment-618182237
ResolveCustomOpMatMul
- no, developed to support old version of TF
- no
- https://github.com/Samsung/ONE/issues/1854
ResolveCustomOpMaxPoolWithArgmax
- no
- no
- https://github.com/Samsung/ONE/issues/7229
- https://github.com/Samsung/ONE/blob/3744b4a4721b7371cdb6ad41b0482ca5fb010362/compiler/luci/pass/src/ResolveCustomOpMaxPoolWithArgmaxPass.cpp#L830
ShuffleWeightTo16x1Float32
- no
- yes, "better latency for aarch64 using neon kernel"
  - this adds one more WeightsFormat of fully connected. However, npu currently seems to use only default WeightsFormat
- https://github.com/Samsung/ONE/issues/4804
ReplaceNonConstFCWithBatchMatMul
- no
- no, but this is from npu requirement, I believe. In case of ONERT, it may degrade the latency because FC is converted into many ops.
- https://github.com/Samsung/ONE/blob/3052fdbd5ae847cfe86daf94725f9559687d9b1a/compiler/luci/pass/src/ReplaceNonConstFCWithBatchMatMulPass.cpp#L67
SubstitutePackToReshape
- no
- no
- https://github.com/Samsung/ONE/blob/df1e9f55d97ba2233174ece7bb919ee0097a7aa6/compiler/luci/pass/src/SubstitutePackToReshapePass.cpp#L94
SubstitutePadV2ToPad
- no
- no
- https://github.com/Samsung/ONE/issues/7610
SubstituteSqueezeToReshape
- no
- no
- https://github.com/Samsung/ONE/issues/5830
SubstituteStridedSliceToReshape
- no
- no
- https://github.com/Samsung/ONE/issues/7232
SubstituteTransposeToReshape
- no. (from Onnx req)
- no
- https://github.com/Samsung/ONE/issues/5703
TransformMinMaxToRelu6Pass
- no (from Onnx)
- no
- https://github.com/Samsung/ONE/issues/5599
TransformMinReluToRelu6Pass
- no (from onnx)
- no
- https://github.com/Samsung/ONE/issues/6714
ExpandBroadcastConst
- no
- yes or no, for npu. const become bigger. not good for ONERT. This increases memory consumption of ONERT.
SparsifyTensorPass
- tf only?
- ??
- https://github.com/Samsung/ONE/issues/3597.
- This should be treated as a user-provided option
ConvertNCHWToNHWC
- yes, onnx only for model that receives NCHW format input
- no
- https://github.com/Samsung/ONE/issues/5401
- This removes Transpose at the beginning of onnx model
- --nchw_to_nhwc_input_shape, nchw_to_nhwc_output_shape are meaningful only when convert_nchw_to_nhwc is set

Another ref: check https://github.com/Samsung/ONE/issues/4353 - which was req from ONERT check grouping issue https://github.com/Samsung/ONE/issues/5784

There is an issue in my internal journal repo, which has a bit more link inside internal data.

Jun 30 '22 08:06 hyunsik-yoon

Jun 30 '22 08:06 seanshpark

Lenged
- -: don't care
- o: should be default
- x: should NOT be default

option	onnx	tflite	ONERT	npu
FusePreActivationBatchNorm	x	x	x	x
MakeBatchNormGammaPositive	x	x	x	x
ReplaceMulAddWithDepthwiseConv	-	-	x	o
ReplaceSubWithAdd	-	-	x	o
ExpandBroadcastConst	-	-	x	o
RemoveQuantDequantSeq	-	-	x	x
RemoveFakeQuant	-	-	x	x
ShuffleWeightTo16x1Float32	x	x	x	x
ReplaceNonConstFCWithBatchMatMul	-	-	x	o
SparsifyTensorPass	x	x	x	x
ConvertNCHWToNHWC	see below	x	-	-
others	o	o	o	o

Regarding ConvertNCHWToNHWC:

if input is ONNX with NCHW input (rank == 4 && transpose with specific perm surrounding some op (e.g., conv2d))
- if input data is NCHW? ==> ConvertNCHWToNHWC is default
- if input/out data is NHWC? ==> ConvertNCHWToNHWC,, nchw_to_hnwc_input_shape, nchw_to_hnwc_output_shape are default

Q. How can we handle a weird Onnx model that has 2 inputs? What if one is NCHW input and another is NHWC input? A. Current onecc also cannot handle this. It changes all input layout format.

(updated) I remvoed ExpandBroadcastConst from the default list. (details in one-vscode-triv repo issue 93.

Jul 01 '22 06:07 hyunsik-yoon

Some thought:

CLI could be something like the following:
- $ one-init a.tflite --model_type=tflite --backend=npu
  - --model_type is optional
- when model type is onnx with image input, two more mandatory args could be provided:
  - $ one-init a.onnx --model_type=onnx --input_data_layout=NCHW --output_data_layout=NCHW --backend=npu
How about put all ops at the last row (others) in the above table into -O1 ?
For npu-only optimization info, where should we put this info?
- location
  - inside a code of one-init vs as a data file read by one-init?
  - I prefer data file because this file has no dependency with ONE ver and can be deployed separately
  - in future, we could consider python script, e.g., def getDefaultOptionsNpu(), when some computation is necessary
- what repo can we store this file?
  - Is it OK to store this file in ONE as a name of npu code name?
  - Or... how about some naming like trix2.json? Note that end user won't enter --backend==trix2.
  - Or we can make another repo that has this data file. Put this file when make internal package. (Such repo could store info that maps npu code name to npu tool executable name....)

Jul 01 '22 07:07 hyunsik-yoon

In https://github.com/Samsung/ONE/issues/9369#issuecomment-1172005559, RemoveQuantDequantSeq and RemoveFakeQuant should not be turned on by default (even for npu). They change the semantics of a model.

Those options were implemented for experiment. I think we should not expose those options to users (CC @seanshpark )

Jul 04 '22 00:07 jinevening

@jinevening Thanks. Table was updated.

Jul 04 '22 02:07 hyunsik-yoon

Let's dig into import options now.

import options

one-import-tflite

input path and output path is all we need. they can be default options.

$ ./one-import-tflite -h
usage: one-import-tflite [-h] [-v] [-V] [-C CONFIG] [-i INPUT_PATH]
                         [-o OUTPUT_PATH]

command line tool to convert TensorFlow lite to circle

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -V, --verbose         output additional information to stdout or stderr
  -C CONFIG, --config CONFIG
                        run with configuation file

converter arguments:
  -i INPUT_PATH, --input_path INPUT_PATH
                        full filepath of the input file
  -o OUTPUT_PATH, --output_path OUTPUT_PATH
                        full filepath of the output file

$ ./one-import-tf
usage: one-import-tf [-h] [-v] [-V] [-C CONFIG] [--v1 | --v2]
                     [--graph_def | --saved_model | --keras_model]
                     [-i INPUT_PATH] [-o OUTPUT_PATH] [-I INPUT_ARRAYS]
                     [-s INPUT_SHAPES] [-O OUTPUT_ARRAYS]
                     [--save_intermediate]
one-import-tf: error: the following arguments are required: -i/--input_path -o/--output_path

one-import-tf

If model is --model_format graph_def, input_array, input_shape, output_array should be mandatory
we should know the following to make default options:
- we should be able to know whether model format is graph_def. If so, the following should be also found to make complete default list.
  - model inputs
  - model input shapes
  - model outputs
- we should be able to know which version of tensorflow (v1 or v2) is used to make model?

https://github.com/Samsung/ONE/blob/c6f0a82d287d8b3c3c975f94b7dfc48901a1fa0c/compiler/tf2tfliteV2/tf2tfliteV2.py#L154-L159

https://github.com/Samsung/ONE/blob/c6f0a82d287d8b3c3c975f94b7dfc48901a1fa0c/compiler/tf2tfliteV2/tf2tfliteV2.py#L141-L144

ref:

https://github.com/Samsung/ONE-vscode/pull/740#discussion_r885229372
https://github.com/Samsung/ONE/issues/9161#issue-1244926748
https://github.com/Samsung/ONE-vscode/issues/693#issuecomment-1135461134,
https://github.com/Samsung/ONE/pull/9160/files#diff-4c7846f58b880ee323ea7d5735e3285f871f308e0d870ece68191ccc39a83aedR33

one-import-onnx

default options can be decided (without difficulty, probably)

https://github.com/Samsung/ONE/issues/9161

[-I INPUT_ARRAYS] [-O OUTPUT_ARRAYS] should be removed from help message

$ ./one-import-onnx  -h
...
usage: one-import-onnx [-h] [-v] [-V] [-C CONFIG] [-i INPUT_PATH]
                       [-o OUTPUT_PATH] [-I INPUT_ARRAYS] [-O OUTPUT_ARRAYS]
                       [--model_format MODEL_FORMAT]
                       [--converter_version CONVERTER_VERSION]
                       [--save_intermediate]

command line tool to convert ONNX to circle

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -V, --verbose         output additional information to stdout or stderr
  -C CONFIG, --config CONFIG
                        run with configuation file
  --save_intermediate   Save intermediate files to output folder

converter arguments:
  -i INPUT_PATH, --input_path INPUT_PATH
                        full filepath of the input file
  -o OUTPUT_PATH, --output_path OUTPUT_PATH
                        full filepath of the output file
  -I INPUT_ARRAYS, --input_arrays INPUT_ARRAYS
                        names of the input arrays, comma-separated
  -O OUTPUT_ARRAYS, --output_arrays OUTPUT_ARRAYS
                        names of the output arrays, comma-separated
  --model_format MODEL_FORMAT
  --converter_version CONVERTER_VERSION

one-import-bcq

also has same options with one-import-tf.

Jul 04 '22 02:07 hyunsik-yoon

one-quantize

one-quantize provides default options (e.g., --input_dtype float32, --quantized_dtype uint8, --granularity layer, --input_type quantized_dtype, --output_type quantized_dtype, --min_percentile 1.0, --max_percentile 99.0, --mode percentile)
- can we use these as default quantization param?
  - note: there could be variations on default value depending on backends. E.g., some backend may prefer channel over layer. Some provides limitation, e.g., channel and layer for q8 but only channel for q16, etc.
input_data https://github.com/Samsung/ONE/blob/c426200a29e4236c73562331c70e5bf35663c0a8/compiler/one-cmds/one-quantize#L44-L50
We may need to set some option to backend-specific default value.

$ ./one-quantize -h
usage: one-quantize [-h] [-v] [-V] [-C CONFIG] [-i INPUT_PATH] [-d INPUT_DATA]
                    [-f INPUT_DATA_FORMAT] [-o OUTPUT_PATH] [-p]
                    [--input_dtype INPUT_DTYPE]
                    [--input_model_dtype INPUT_MODEL_DTYPE]
                    [--quantized_dtype QUANTIZED_DTYPE]
                    [--granularity GRANULARITY] [--input_type INPUT_TYPE]
                    [--output_type OUTPUT_TYPE]
                    [--min_percentile MIN_PERCENTILE]
                    [--max_percentile MAX_PERCENTILE] [--mode MODE]
                    [--force_quantparam] [--tensor_name TENSOR_NAME]
                    [--scale SCALE] [--zero_point ZERO_POINT]

command line tool to quantize circle model

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -V, --verbose         output additional information to stdout or stderr
  -C CONFIG, --config CONFIG
                        run with configuation file
  -i INPUT_PATH, --input_path INPUT_PATH
                        full filepath of the input circle model
  -d INPUT_DATA, --input_data INPUT_DATA
                        full filepath of the input data used for post-training
                        quantization. if not specified, run with random input
                        data.
  -f INPUT_DATA_FORMAT, --input_data_format INPUT_DATA_FORMAT
                        file format of input data. h5/hdf5 (default),
                        list/filelist (a text file where a file path of input
                        data is written in each line), or dir/directory (a
                        directory where input data are saved)
  -o OUTPUT_PATH, --output_path OUTPUT_PATH
                        full filepath of the output quantized model
  -p, --generate_profile_data
                        generate profiling data

arguments for quantization:
  --input_dtype INPUT_DTYPE
                        input model data type (supported: float32,
                        default=float32). Deprecated (Use input_model_dtype)
  --input_model_dtype INPUT_MODEL_DTYPE
                        input model data type (supported: float32,
                        default=float32)
  --quantized_dtype QUANTIZED_DTYPE
                        data type of output quantized model (supported: uint8,
                        int16, default=uint8)
  --granularity GRANULARITY
                        quantization granularity (supported: layer, channel,
                        default=layer)
  --input_type INPUT_TYPE
                        data type of inputs of quantized model (supported:
                        uint8, int16, default=quantized_dtype). QUANTIZE Op
                        will be inserted at the beginning of the quantized
                        model if input_type is different from quantized_dtype.
  --output_type OUTPUT_TYPE
                        data type of outputs of quantized model (supported:
                        uint8, int16, default=quantized_dtype). QUANTIZE Op
                        will be inserted at the end of the quantized model if
                        output_type is different from quantized_dtype.
  --min_percentile MIN_PERCENTILE
                        minimum percentile (0.0~100.0, default=1.0). Algorithm
                        parameter for calibration. This is valid when
                        calibration algorithm is percentile.
  --max_percentile MAX_PERCENTILE
                        maximum percentile (0.0~100.0, default=99.0).
                        Algorithm parameter for calibration. This is valid
                        when calibration algorithm is percentile.
  --mode MODE           calibration algorithm for post-training quantization
                        (supported: percentile/moving_average,
                        default=percentile). 'percentile' mode uses the n-th
                        percentiles as min/max values. 'moving_average' mode
                        records the moving average of min/max.

arguments for force_quantparam option:
  --force_quantparam    overwrite quantparam (scale, zero_point) to the
                        specified tensor in the quantized model.
  --tensor_name TENSOR_NAME
                        tensor name (string)
  --scale SCALE         scale (float)
  --zero_point ZERO_POINT
                        zero point (int)

if --input_data is not provided by default, one-quantize will use random data and this decrease accuracy.
- option 1) Add comment (with suggestion ID) inside generated cfg file mentioning this
  - ; #suggestion: With no 'input-data', random data will be used and this could decrease accuracy. Consider using input data file.

Jul 04 '22 04:07 hyunsik-yoon

discussed with @lemmaa and @seanshpark

How to handle option that is dependent on backend

option 1.
- use data file that contains backend-specific default option
- with cfg v2 (https://github.com/Samsung/ONE/issues/9192), this data file format would become complex.
option 2.
- like other ONE backend tools, create backendname-init tool (e.g., for foo backend, create foo-init tool)
- one-init first create cfg for ONE. then foo-init will add more default options for foo backend.

Jul 04 '22 10:07 hyunsik-yoon

`one-init`

Let's design CLI.

I added an option into CLI if

the option does not have any default value
without the option, compilation fails

$ /usr/share/one/bin/one-init -h
usage: one-init [-h] [-v] [-i INPUT_PATH] [-t INPUT_TYPE] [-b BACKEND] [--] [COMMANDS FOR BACKEND]

command line tool for generate *.cfg file for the given model

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -V, --verbose         output additional information to stdout or stderr

arguments for generation:
  -i INPUT_PATH, --input_path INPUT_PATH
                        full filepath of the input model file
  -o OUTPUT_PATH, --output_path OUTPUT_PATH
                        full filepath of the output cfg file
  -m MODEL_TYPE, --model_type MODEL_TYPE
                        type of input model: "onnx", "tflite", "tf", "bcq"
                        If this option is not provided, file extension is used to decide model type.
  --tf_graphdef_input_arrays INPUT_ARRAYS 
                        names of the input arrays, comma-separated
                        (argument when input model is TF graphdef)            
  --tf_graphdef_input_shapes INPUT_SHAPES
                        shapes corresponding to --input_arrays, colon-
                        separated (ex:"1,4,4,3:1,20,20,3")
                        (argument when input model is TF graphdef)            
  --tf_graphdef_output_arrays OUTPUT_ARRAYS (argument when input model is TF graphdef)
                        names of the output arrays, comma-separated
                        (argument when input model is TF graphdef)            
  --onnx_input_data_layout LAYOUT
                        layout of input data files (argument when input model is onnx)
                        value could be: "NCHW", "NHWC", "not_image"
                        When input layout is neither NCHW nor NHWC, use "not_image".
  --onnx_output_data_layout LAYOUT
                        layout of output data files (argument when input model is onnx)
                        value could be: "NCHW", "NHWC", "not_image"
                        When output layout is neither NCHW nor NHWC, use "not_image".
  -b BACKEND, --backend BACKEND
                        backend name to use

`trix-init`

Like other ONE tool and backend tools, one-init could call backend-init cmd. let's assume that backend name is "trix".

candidate 1.

$ trix-init -h [-C CONFIG]

arguments for generation:
  -C CONFIG     add default options into ONE configuation file

However, we cannot assume that all backend provider has ini or json lib. So this approach does not seem to good.

candidate 2.

Calling trix-init prints default option to console in json. Note that printing such json does not require json lib.

$ trix-init 
{ 
  "optimize": ["ReplaceMulAddWithDepthwiseConv", "ReplaceSubWithAdd", ..., ],
  "quantize": { "quantized_dtype": "uint8", "granularity": "channel", ...  }
}

Jul 06 '22 10:07 hyunsik-yoon

Discussion with @seanshpark

args for onnx only
- args originated for onnx model are better NOT to included for, e.g., tflite model compilation because
  - adding more args will degrade compilation speed
  - such op sequences rarely appear in tflite -[ ] classify those args and apply them only for onnx model
ConvertNCHWToNHWC -[ ] For onnx model, this needs to be default. Add this option for onnx model
Make cfg for compilation (not for accuracy, memory, latency) at first step
- Related to https://github.com/Samsung/ONE/issues/9313#issuecomment-1165442530
- Accuracy could not be good with some default op (e.g., default arg of one-quantize uses random data, which could lower the accuracy)
- For the first step, focus on cfg file with which compilation will succeed ; for the next step, consider accuracy, memory, and latency
comment inside cfg
- one-init can generate useful comment for user
- Consider how ONE-vscode cfg editor can do the following: (/cc @llFreetimell)
  - show comment
  - save comment when user press Ctrl-S
grouping args of one-optimizes
- We can start tasks related to grouping args after releasing one-init if dev date for releasing one-init is not long enough

Jul 13 '22 01:07 hyunsik-yoon

args originated for onnx model are better NOT to included for, e.g., tflite model compilation

Let's list onnx-only args. These will be removed when model is not onnx.

`one-optimize` arg from onnx (WIP)

ForwardReshapeToUnaryOp
- https://github.com/Samsung/ONE/issues/5848
FuseBatchNormWithDwConv
- https://github.com/Samsung/ONE/issues/5747
FuseMeanWithMean
- https://github.com/Samsung/ONE/issues/7335
- https://github.com/Samsung/ONE/pull/7470 -FuseTransposeWithMean
- https://github.com/Samsung/ONE/issues/7335#issuecomment-893127343
from https://github.com/Samsung/ONE/issues/7350
- RemoveRedundantReshape
- RemoveRedundantTranspose
- RemoveUnnecessaryReshape
- RemoveUnnecessarySlice
- RemoveUnnecessarySplit
- SubstituteTransposeToReshape
  - https://github.com/Samsung/ONE/issues/5401
- SubstituteSqueezeToReshape
- FoldSparseToDense
- FoldAddV2
- FoldCast
SubstitutePadV2ToPad
- https://github.com/Samsung/ONE/issues/7477
SubstituteSqueezeToReshape
- https://github.com/Samsung/ONE/issues/5830
SubstituteStridedSliceToReshape
- https://github.com/Samsung/ONE/issues/7232#issuecomment-884617586
- GRU model was an onnx model
TransformMinMaxToRelu6Pass
- https://github.com/Samsung/ONE/issues/5599 ("onnx-tf generates the above pattern for Relu6")
TransformMinReluToRelu6Pass
- https://github.com/Samsung/ONE/issues/6616#issuecomment-828862547
ConvertNCHWToNHWC
- https://github.com/Samsung/ONE/issues/5401

suspect (these could be introduced (심증) for onnx but I couldn't find any info):

RemoveUnnecessaryStridedSlice
- https://github.com/Samsung/ONE/issues/5757
SubstitutePackToReshape
- https://github.com/Samsung/ONE/issues/5180

Jul 13 '22 04:07 hyunsik-yoon

@seanshpark @jinevening Could you briefly check https://github.com/Samsung/ONE/issues/9369#issuecomment-1182739358? Thanks in advance!

Jul 13 '22 06:07 hyunsik-yoon

The options in https://github.com/Samsung/ONE/issues/9369#issuecomment-1182739358 were introduced for onnx models, but they can be used also for other frontends (tf, tflite).

Among the list, I think onnx-only option is only ConvertNCHWToNHWC.

Jul 13 '22 06:07 jinevening

Could you briefly check https://github.com/Samsung/ONE/issues/9369#issuecomment-1182739358?

but they can be used also for other frontends (tf, tflite).

I agree on this but in the first step, for some safety, I prefer put these only to ONNX models. And in the second step, with some experience and testing, we can put to common.

Jul 13 '22 07:07 seanshpark

@jinevening

The options in https://github.com/Samsung/ONE/issues/9369#issuecomment-1182739358 were introduced for onnx models, but they can be used also for other frontends (tf, tflite).

I agree.

What @seanshpark has concern about was that the patterns for these args might rarely occur in tflite models. So we thought that adding those args could increase compilation time for tflite models but show small impact on optimization.

Like @seanshpark suggested, how about adding those args into the second step someday? Meanwhile, inside cfg files for tflite models, I think that we can add some comment about these args so that users can easily turn on some of these options.

Jul 13 '22 08:07 hyunsik-yoon

I get it. Let's go with that option :)

Jul 13 '22 08:07 jinevening

Let me close this and create another issue to implement one-init.

Jul 18 '22 05:07 hyunsik-yoon

I got the following error:

$ onecc -C gen-from-one-init-for-tflite.cfg
usage: one-optimize [-h] [-v] [-V] [-C CONFIG] [-p]
                    [--change_outputs CHANGE_OUTPUTS] [-i INPUT_PATH]
                    [-o OUTPUT_PATH] [--convert_nchw_to_nhwc]
...
                    [--fuse_activation_function] [--fuse_instnorm]
                    [--replace_cw_mul_add_with_depthwise_conv]
                    [--remove_fakequant] [--remove_quantdequant]
...
                    [--remove_unnecessary_split]
                    [--replace_non_const_fc_with_batch_matmul]
                    [--resolve_customop_add] [--resolve_customop_batchmatmul]
...
                    [--transform_min_relu_to_relu6]

one-optimize: error: the following arguments are unrecognized: replace_sub_with_add

The error msg says that replace_sub_with_add is not recognized. Hmm..

replace_sub_with_add is not in constant.py. I am not sure if this is intentional or a bug.

Jul 18 '22 08:07 hyunsik-yoon

I am not sure if this is intentional or a bug.

Bug -_-; If you like, you can add this to one-cmds :)

or... @llFreetimell was this intentional ?

Jul 18 '22 09:07 seanshpark

I added replace_sub_with_add into /usr/share/one/bin/onelib/constant.py and now I got model-opt.circle. However one-quantize doesn't seem to work.

/usr/share/one/bin/one-quantize -V -i ./models/model-opt.circle -o ./models/model-q.circle --quantized_dtype uint8 --granularity channel

Jul 18 '22 09:07 hyunsik-yoon

However one-quantize doesn't seem to work.

ping @jinevening

Jul 18 '22 10:07 seanshpark

replace_sub_with_add is not in constant.py. I am not sure if this is intentional or a bug

I think it's simple mistake of @mhs4670go @.@ After I finished the implementation of replaceSubWithAdd, constant.py was created...!

Jul 18 '22 10:07 llFreetimell

ONE ONE copied to clipboard

Analyze onecc options

one-optimizer options

Format

Optimization passes

import options

one-import-tflite

one-import-tf

one-import-onnx

one-import-bcq

one-quantize

How to handle option that is dependent on backend

one-init

trix-init

candidate 1.

candidate 2.

Discussion with @seanshpark

one-optimize arg from onnx (WIP)

ONE
ONE copied to clipboard

`one-init`

`trix-init`

`one-optimize` arg from onnx (WIP)