[ONE-Dockerizing] shell script related to onecc-docker

[Seunghui98]

What

To implement onecc-docker, we created a shell script for creating a docker image file, building an image, and executing a container.

Why

Regarding #8232 issue, we are going to do Dockerizing so that more users can easily use onecc Related Issue : #9714, #9712

How

Work flow

• Get the version of onecc installed on the Host PC
• Get version for latest onecc
  • Output exception message when request fails
• If the latest onecc is installed on the Host PC are the same, the input onecc-cmd is executed.
• If the latest one is not installed, onecc-docker containers runs.
  • If one image does not exist, generates onecc-docker image file through write_dockerfile() and build onecc-docker image.
  • Run the onecc-docker container.

How to test

I currently proceeded with Dockerizing in the following way. Since onecc-docker was mounted as /home/, we tested it to create a circle file in /home/test.

summary

1. Import TensorFlow Model for Dockerizing and write Configure File.
2. Write a Dockerfile to create a Docker image and run the container.
   • When onecc is finished running, the container is terminated and deleted.
3. Check that circle file and nnpackage have been created.

Import Tensorflow model for testing

First, I created the path '/home/test/' for Linux (wsl) and downloaded the model file to test. After downloading inception_v3, the Configure file was created.

# /home/test/
/home/test$ wget https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/inception_v3_2018_04_27.tgz

/home/test$ tar -xvf inception_v3_2018_04_27.tgz

/home/test$ vi onecc.template.cfg

/home/test$ tree
.
├── inception_v3.pb
├── inception_v3.tflite
├── inception_v3_2018_04_27.tgz
├── labels.txt
└── onecc.template.cfg

• onecc.template.cfg

  [onecc]
  one-import-tf=True
  one-import-tflite=False
  one-import-bcq=False
  one-import-onnx=False
  one-optimize=True
  one-quantize=False
  one-parition=False
  one-pack=True
  one-codegen=False
  
  [one-import-tf]
  input_path=inception_v3.pb
  output_path=inception_v3.circle
  input_arrays=input
  input_shapes=1,299,299,3
  output_arrays=InceptionV3/Predictions/Reshape_1
  converter_version=v1
  model_format=graph_def
  
  [one-optimize]
  input_path=inception_v3.circle
  output_path=inception_v3.opt.circle
  generate_profile_data=False
  
  [one-pack]
  input_path=inception_v3.opt.circle
  output_path=inception_v3_pack
  

run onecc-docker

$ cd ONE/compiler/onecc-docker/
$ chmod +x onecc-docker 
$ ./onecc-docker

Execution Results

This is my execution result. The circle file and nnpackage have been created well, and I don't think there is any problem with my method so far.

Docker container was also successfully removed.

• Test results

/home/test$ tree
.
├── inception_v3.circle
├── inception_v3.circle.log
├── inception_v3.opt.circle
├── inception_v3.opt.circle.log
├── inception_v3.pb
├── inception_v3.tflite
├── inception_v3_2018_04_27.tgz
├── inception_v3_pack
│   └── inception_v3.opt
│       ├── inception_v3.opt.circle
│       └── metadata
│           └── MANIFEST
├── inception_v3_pack.log
├── labels.txt
└── onecc.template.cfg

3 directories, 12 files

/cc @Samsung/ootpg

[jyoungyun]

I think this issue contains the final goal. And you're going to split this issue up, right? :)

[seanshpark]

Since onecc-docker was mounted as /home/, we tested it to create a circle file in /home/test.

I'm not sure this is acceptable. In general, touching folders in /home manually is not normal. @jyoungyun , any comments?

[Seunghui98]

@jyoungyun

Yesterday, you reviewed the new PR separately in two. It was ambiguous to divide it into two because two functions were connected within one shell file and existed within one file. I posted the shell as PR, is it okay?

• Related PR : #9722

[jyoungyun]

Since onecc-docker was mounted as /home/, we tested it to create a circle file in /home/test.

I'm not sure this is acceptable. In general, touching folders in /home manually is not normal. @jyoungyun , any comments?

@Samsung/ootpg_docker Um, do we have to create a special directory for test in user home directory? I also think it is not nature.

[jyoungyun]

Yesterday, you reviewed the new PR separately in two. It was ambiguous to divide it into two because two functions were connected within one shell file and existed within one file. I posted the shell as PR, is it okay?

Sure. I like this change better. :)

[holawan]

@seanshpark @jyoungyun When we used that route, we didn't know how to mount it with volume, but it was just for testing. I'm sorry that the contents of the test went in by mistake.

Here's how we do it now. When the docker container runs, share /home:/homeas a volume and set the user's current location to working directory so that the result is produced where the user runs onecc-docker.

Here's an example.

docker run --rm --name {container_name} -v /home:/home --workdir {user_pwd} {docker_image} {user_command}

How about this way?

[seanshpark]

/home:/home

I'm not used to Docker. Is this usual case? Why no use /home/myaccount:/home/myaccount , for $HOME is /home/myaccount ?

[jyoungyun]

Here's an example.

docker run --rm --name {container_name} -v /home:/home --workdir {user_pwd} {docker_image} {user_command}

How about this way?

What about using -v ${HOME}:${HOME} -e HOME=${HOME} -w $(pwd) ?

The reason why we mount ${HOME} is that the several files described in cfg file can be relative path. To prevent the user model of the relative path from not being accessed, we mount ${HOME} environment.

[seanshpark]

The reason why we mount ${HOME} is that ...

I was curious why mount host into Docker machine... it was for accessing files :)

This restrion gives up that we cannot compile models in other folders, like /tmp. I think we have to go with this as usually users work with inside somewhere in $HOME

[holawan]

@jyoungyun @seanshpark I understand what you're saying. There seems to be no reason to use '/home:home'. I will modify the code by reflecting PR and current issues. Thank you.

[holawan]

I think this issue contains the final goal. And you're going to split this issue up, right? :)

@jyoungyun, I have a question regarding the mansion. Since the part included in the issue is our next step, it doesn't have to be included, right?

Whether to install onecc on the host PC.

In the next PR, may I exclude this part?

[jyoungyun]

@jyoungyun, I have a question regarding the mansion. Since the part included in the issue is our next step, it doesn't have to be included, right?

I think it is good to manage the whole project issue and individual issues separately. This issue is meaningful because it contains a big picture of the whole process. Apart from this issue, it would be nice to have more individual issues that focus on what is currently implementing.

In the next PR, may I exclude this part?

Nope. :) This issue is enough. Please create an issue that focus on what is currently implementing.

[jyoungyun]

SSAFY Project was finished. I will close it. :)