[ONE-Dockerizing] Introduce ONE Dockerizing

[holawan]

What

Let’s introduce ONE Dockerizing.

Why

Currently, onecc is only supported on Ubuntu 18.04 and 20.04 (not offially).

Therefore, it is not easy for people to use onecc in other environments.

Our ONE-Dockerizing will use Docker to make onecc easy for users to use in other environments. The ultimate goal will be to use onecc with Docker on Windows, Mac, etc., but we are currently working on Linux, and our goal is to do ONE-Dockerizing on all Linux environments.

Scenario

System Scenario

1. Receive commands entered by the user.
2. Verify that onecc-docker can be run in the user's environment (existence of docker, installation of python3, etc.)
3. Run Docker Container
   • When Docker Image is not present
     • Create the docker image
     • Run Docker Container
   • When Docker Image is present
   • Run the docker container by specifying the volume as '/home:/home' and by specifying the location where the current user is running as WORKDIR.
     • docker run --rm --name {container_name} -v /home:/home --workdir {user_pwd} {docker_image} {user_command}
4. Create circle files and nnnpackage using the .cfg and TensorFlow models shared as volumes inside the docker container and provide them to the user.

User Scenario

• Create a *.cfg and TensorFlow Model file(.pb, .tfilte ..) containing the option you want to perform related to the "onecc"

• The user runs the onecc-docker. (including the flag desired by the user)

  • onecc-docker -C *.cfg

• The user will get the desired result. Below is an example.

  • version
  • circlefile,nnpackages
  • help message

TODO

We will first run Docker-related tasks to run "onecc-docker". Because this requires a Dockerfile to create a Docker Image, we will create it. Second, we're going to write an "onecc-docker" shell script and run Docker on the system. Finally, we would like to provide Debian Package with "onecc-docker".

I have summarized and will update what I need for this work below.

• [x] Create Dockerfile
  • [x] Make Docker container run successful with commands entered by the user
    • [x] -I, -C, -O,..
• [x] Create onec-docker shell script
  • [ ] Check for docker installation script
  • [ ] Check for python3 installation script
  • [ ] Check for onecc installation script
  • [x] run docker if docker image exists, and create a script to build if not.
• [x] Create onecc-docker Debian Package

related issue : #8232

/cc @Samsung/ootpg

[seanshpark]

About the scenario, this is not that important but I'd like to mention this;

• It would be better to separate scenario in user point of view from internal implementation
• I think user would act like
  • create cfg file like mymodel.cfg with mymodel.pb model
  • run onecc-docker -C mymodel.cfg
  • user gets mymodel.circle
• other things are done inside our system which canbe multiple of solutions

[seanshpark]

About todo...

• I actually don't understand fully about internal implementation details
• I think you first do describe how you are going to make it done before listing TO DOs.

[holawan]

@seanshpark Thank you for your opinion. :)

About the scenario, this is not that important but I'd like to mention this;

I will add it considering the scenario of the user's position.

About todo...

And I will revise Todo by referring to other issues.

[jyoungyun]

I think you first do describe how you are going to make it done before listing TO DOs.

The final output results

onecc-docker debian package

• This team will generate a onecc-docker debian package including several scripts to run onecc and Dockerfile to build a docker image.

User scenario

1. Install onecc-docker debian package
2. Run onecc-docker script
   1. If one-compiler was installed before
      • Run onecc through onecc-docker script
   2. No installed one-compiler
      • Build docker image that contains the one-compiler
      • Run onecc through the built docker image

Would it be helpful to understand if I put more explanations like this?

/cc @seanshpark

[seanshpark]

Install onecc-docker debian package

Q) so the user do not or cannot install one-compiler debian package?

• like in the future, say for mac osx, if we (or someone else) provide onecc-docker.dmg somehow, then the user can run onecc inside the Docker through onecc-docker ?

[seanshpark]

If one-compiler was installed before

Q) is this line one-compiler debian package install in native or is it through Docker image?

No installed one-compiler

Q) Seems this is native one-compiler debian package. Am I right? Q) After this, there will be onecc running inside Docker. If the user installs one-compiler then there will be onecc inside Docker and in the Native Which may be in different version. Have any plans for version mismatch? Q) Is there any selection option to choose which onecc(Docker or Native) if both installed? Or just run Docker in higher priority ? Q) Can we fix the senario that one-docker will ONLY run Docker image for first implementation for simplicity?

[jyoungyun]

Q) so the user do not or cannot install one-compiler debian package?

You can install one-compiler debian package regardless of whether onecc-docker debian package is installed.

• like in the future, say for mac osx, if we (or someone else) provide onecc-docker.dmg somehow, then the user can run onecc inside the Docker through onecc-docker ?

Yes, that's what we're ultimately trying to do. :) (Although this SSAFY does not create a package for mac..)

Q) is this line one-compiler debian package install in native or is it through Docker image?

It means that one-compiler debian package is installed in your file system.

Q) Seems this is native one-compiler debian package. Am I right?

Yes, you're correct. :)

Q) After this, there will be onecc running inside Docker. If the user installs one-compiler then there will be onecc inside Docker and in the Native Which may be in different version. Have any plans for version mismatch?

And so, we consider the version argument from onecc-docker script file. For now, we don't care about the version. So if the onecc is installed in Native, we will use onecc. And if there is no installed onecc, we will use the latest onecc inside Docker image. (Because we use the latest one-compiler package when we build the docker image.)

Afterward, if we support the version, we will input the version number. And then if the exactly matched one-compiler is already installed, we will use it in Native. However if the installed version is not matched from user inputs, we will use onecc in docker image. If user does not input any version number, the native onecc has a higher priority than onecc in the latest docker image.

Q) Is there any selection option to choose which onecc(Docker or Native) if both installed? Or just run Docker in higher priority ?

The native onecc has a higher priority.

Q) Can we fix the senario that one-docker will ONLY run Docker image for first implementation for simplicity?

Could you explain a little more? I can't catch your intention. :)

[seanshpark]

Could you explain a little more? I can't catch your intention. :)

by the scenario, onecc-docker can either run onecc in Native installed or the Docker version. for the first implementation, let's just focus on running ONLY in the Docker version. do not care if there is Native one-compiler package installed or not.

after this works OK and stable, we can revise to check if Native version is installed... and so one with more complicate scenarios until the time permits.

[jyoungyun]

by the scenario, onecc-docker can either run onecc in Native installed or the Docker version. for the first implementation, let's just focus on running ONLY in the Docker version. do not care if there is Native one-compiler package installed or not.

Oh, that's a good idea. I think this approach will be a clear and safe design. What do you think of this?

/cc @Samsung/ootpg @Samsung/ootpg_docker

[holawan]

Oh, that's a good idea. I think this approach will be a clear and safe design. What do you think of this?

I think it means that the scenario needs to be modified.

Are you saying that we should focus on running onecc-docker using docker without checking whether the user's native one-compiler is installed (version mismatch, installation, etc.)?

• When running onecc-docker, provide circle file or nnpackages to the user

If I understand correctly, I will attach the skeleton code and give you additional PR soon :)

Thank you for your opinion :)

[seanshpark]

I think it means that the scenario needs to be modified.

In another words, if we are at point A and want to go to point B and there are several intermediate points C and D and so on. These intermediate points are hidden :) Point B is our final goal. We are to select how to reach B. Scenario itself isn't changed but we're defining which intermediate points to visit.

I proposed first without selection to run onecc-docker only to look at Docker, not the natively installed one-compiler. This will make logic simple and implementation simple and test simple.

When related inplemention have landed, we can then change the code to look ALSO to one-compiler installed. And add testes for ONLY CHANGED parts.

"Why should I follow this suggestion?" Please take this in mind in this period and find the "why?"

And you don't have to follow this way, just your choice.

[jyoungyun]

I think it means that the scenario needs to be modified.

Are you saying that we should focus on running onecc-docker using docker without checking whether the user's native one-compiler is installed (version mismatch, installation, etc.)?

Yes, that's exactly what I said. :)

We can break down our plan like this:

• [ ] Introduce new onecc-docker script files and Dockerfile
  • [ ] Packaging to a debian package
    • [ ] Upload somewhere to share a generated debian package with ONE-vscode
  • [ ] Support to run a native onecc
  • [ ] Configure a test framework (if you want)
    • [ ] Add test cases

Items at the same level are able to proceed in parallel. And child items are dependent on parent item. It would be nice to organize items that can be implemented in parallel or series.

It is just my suggestion, all plans are up to @Samsung/ootpg team. :)

[holawan]

@jyoungyun, the code may not work properly because there is no dockerfile in the current PR(#9772) . Can you give me your opinion?

[jyoungyun]

@jyoungyun, the code may not work properly because there is no dockerfile in the current PR(#9772) . Can you give me your opinion?

No, problem. Other commit will be merged soon. So there is nothing to worry about just because it does not work with One PR. :)

[holawan]

https://github.com/Samsung/ONE/pull/9737#discussion_r975311818 Docker image build takes a certain amount of time. Therefore, the user must be able to recognize that the docker image is currently being built.

In this regard, I have considered two solutions.

1. Print the docker image build process as a log.

2. Print a progress bar during docker image build.

Please let me know your preferred method. I will attach the gif.

1. build log

2. progress bar I did two simple implementations.

• Simple spinner One is a self-implemented progress. It doesn't matter if it's library dependent, but it's less pretty.

• Progress Bar The second is using python library "alive_progress", which is related to library dependence, but looks a little more lively and pretty.

If you think it's reasonable to put a progress bar, which one do you think is better between the two methods? @jyoungyun

[jyoungyun]

@seanshpark @mhs4670go @lemmaa

To create a docker image takes a long time. So @Samsung/ootpg team suggested a couple of ways. ~~Please give us your opinion on which of the above proposed methods is better. :)~~ Please vote for the one you like among the methods below. :)

@holawan @Seunghui98

I prefer 2. progress bar. Because showing the docker log is difficult and unnecessary for the user. I like both of 2. progress bar ways, but if I have to choose between the two,,,, I like the progress Bar because it is more visualized. :)

[jyoungyun]

1. build log

[jyoungyun]

• Simple spinner One is a self-implemented progress. It doesn't matter if it's library dependent, but it's less pretty.

[jyoungyun]

• Progress Bar The second is using python library "alive_progress", which is related to library dependence, but looks a little more lively and pretty.

[seanshpark]

As I know this feature is WIP and maybe till test is introduced later. For development, I prefer order of (1) show log to look at the problems if any (2) show progress bar. Network behind firewall with proxy environment is not that good.

If this is shown in VSCode extension, (1) is not user friendly. Thus, it would be better if this is possible

• show log with console or user normal command
• show progresss bar in VSCode extension

Anyway supporting as like will take time and time SSAFY team may not have much. Conclusion; go with eaither case, (3) is not preferend, and add some TODO comments.

[jyoungyun]

@seanshpark

Do you agree to apply the (2) case? If so, she will apply it. And for the record, I will ask her to upload the code developed for (3) as draft for her. The code will not be reflected, so it will be closed..

[seanshpark]

Do you agree to apply the (2) case? If so, she will apply it.

I wrote "go with eaither case, (3) is not preferend, and add some TODO comments." Currently I'm not aware of how the UX will come up with command lines and in VSCode. I hope there are some comments in the code.

[jyoungyun]

@Seunghui98 Could you apply (2) case with TODO comments? :)

[lemmaa]

It seems a bit late, but just to leave a comment, I'm basically the same as @seanshpark's comment. Personally, (3) is not preferred over (2) because unnecessary dependency occurs without providing additional information.

[jyoungyun]

SSAFY Project was finished. I will close it. :)