Upload docker image to dockerhub

[sornian]

https://hub.docker.com/u/sjtucaocao

Images in docker hub is not the new version

I try to build base.docker but our country can't visit Google any website

Could you please upload the new image to docker hub

Please

[caoccao]

Thank you for raising this issue. I have been trying every week, however I have failed to do so successfully over the past 2 weeks because of the VPN inability 😭.

[sornian]

Most developers only use javet or edit the javet source code without directly modifying the source code of V8 use the v8 compiled file is enough could we upload the v8 compile file to github?

[caoccao]

I don't think that is a good idea. Those binary files could reach tens of GBs so that cloning this project would be impossible.

[amithgeorge]

@caoccao we could setup the github actions, such that any tagged commit gets its docker image built and push to dockerhub. I could do a PR for that if needed.

[caoccao]

we could setup the github actions, such that any tagged commit gets its docker image built and push to dockerhub. I could do a PR for that if needed.

Here are the challenges for your reference.

1. Building the base image takes 1-2 hours. That would breach Github action execution limit (1 hour) and be killed.
2. Uploading the base image to docker hub takes another 1-2 hours as the image size is 10-15GB.

If you could resolve these challenges, I would be happy to merge your PR.

By the way, I'm using an VPN to get out of China mainland. That VPN charges me by the data volume. It would take 20+GB per base image. That's a heavy financial burden to me.

[amithgeorge]

Ok. I may have misunderstood something. What is the purpose of the uploaded docker image? From the dockerfile in the repo, it looks like the image is used to build the Javet binaries. Is the build image primarily for people wanting to contribute to Javet development? And not meant for users of Javet?

[amithgeorge]

Keeping aside the question of why is the image needed, we can tackle the concerns raised,

Building the base image takes 1-2 hours. That would breach Github action execution limit (1 hour) and be killed.

From the documentation for limits on actions, https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration#usage-limits

Job execution time - Each job in a workflow can run for up to 6 hours of execution time. If a job reaches this limit, the job is terminated and fails to complete.

Maybe the 1 hour limit you mentioned was an older limit? If I have missed something, please correct me.

Uploading the base image to docker hub takes another 1-2 hours as the image size is 10-15GB.

Do we need the source code to be present in the final built image? Docker has the concept of multi-stage builds, where we use one image to build the compiled files and then copy the compiled files to another image. The resultant final image is much smaller. As an example, using multi stage builds, a user was able to reduce the size of the V8 image from 5GB to 0.47GB. Reference

We can apply the same concept to building Node (though I am not sure why are we building Node instead of installing it). To speed up the overall build, we can choose to build the V8 image independently from the Node image and refer both in the multi-stage build for Javet base. It won't be trivial to do, but if you could help me understand what is happening and what is actually needed (ie only compiled binaries and no source), then I can attempt something.

[caoccao]

Thank you for the help.

Ok. I may have misunderstood something. What is the purpose of the uploaded docker image? From the dockerfile in the repo, it looks like the image is used to build the Javet binaries. Is the build image primarily for people wanting to contribute to Javet development? And not meant for users of Javet?

It's for building and testing every commit.

I think I was perceived by some document about the 1 hour limit. Thank you for correcting me.

Regarding shrinking the image, it's tricky because Javet references the source code directly as well as the libraries. That implies the whole build environment, source code and .obj files are supposed to be in the image. You may take a try and find out how much you may drop. The largest image layer is 5+GB so that the connection to docker hub drops so frequently. Especially, as I'm behind the VPN, I haven't pushed such image successfully over the past few months. I doubt how stable the connection between github and docker hub.

[amithgeorge]

Reading through the Dockerfile and build scripts for linux, it looks like for the V8 build of Javet it only needs the V8 source and compiled folders. For the Node build of Javet, does it only need the Node source and compiled folders, or does it need the V8 ones as well?

If they are independent, then we can split the images, reducing their size further. However, if the Node build requires the V8 bits as well, we can base the Node image off the V8 image. The idea is instead of one huge 15gb image, we might have three images of 5gb each (these numbers are assumptions) - a V8 image, a Node image based off the V8 image, and a Javet image based off the Node image. The V8 and Node image will change maybe once every 4-6 weeks as per their release cycles. The 3rd Javet image may change more frequently, though that might be much smaller in size. Just ideas in my head atm.

[caoccao]

You are right. The dockerfiles can be layered as the diagram below. You are welcome making changes to the dockerfiles and workflow to achieve that goal. Thank you.

[amithgeorge]

Nice.

I had some trouble setting up an Ubuntu VM on my Windows 10 + WSL2 desktop. Need to figure out the details. Once that is done, I should be able to actually build the existing docker images. Give me a few days to figure things out.

[caoccao]

Nice.

I had some trouble setting up an Ubuntu VM on my Windows 10 + WSL2 desktop. Need to figure out the details. Once that is done, I should be able to actually build the existing docker images. Give me a few days to figure things out.

That will be great. Thank you very much.

[sornian]

嘿嘿

[amithgeorge]

@caoccao In my excitement for solving a technical challenge, I think I missed out on understanding the purpose of the image. Let me state my current understanding and ask clarifying questions.

The latest image on https://hub.docker.com/r/sjtucaocao/javet/tags is tagged 1.1.5. This image is created from the dockerfile docker/linux-x86_64/base.Dockerfile. You mentioned in an earlier message that the purpose of the docker image is

"It's for building and testing every commit."

I can see in the Github workflow, that this image is used to test the code on every pull request, and to build the jar files and upload them as artifacts.

1. Is that the only place where this docker image is used? Will we (you or any contributor to Javet source) also be using the docker image for local development?
2. If we think about it, the base image only contains the V8 and Node source/binaries. Let's ignore the jar dependencies downloaded using gradle for now. If we consider only the V8 and Node dependencies, then the name of the docker image and it's tag should change. Sample image name and tag - javet-dev:v8-10.3.174.14_node-16.15.1. I feel tagging this way makes it more obvious what the image is meant for, as opposed to naming it javet:1.1.5. I am open to a different naming scheme.

3.1. I understand the desire to cache the dependency jar files using gradle in the base.Dockerfile. I would suggest achieving that purpose in a different way. Let's change the purpose of the docker/linux-x86_64/build.Dockerfile file. Currently as part of building the image, the artifact jar is created. Meaning the image itself has no value, except for performing a side effect. That's why we have this awkward step of running the image and copying the built artifacts.

3.2. Instead we strip the build.Dockerfile of almost all the commands, keeping it bare. We build the image and run the container. During the run, we mount the current source code directory at the path /Javet and provide a shell script to execute. The shell script does the Build JNI and Build Jar steps. The jar is written to the mounted path. Once the container has finished execution, the artifact jars are already available for uploading. The advantage of this approach is, along with mounting the source code, we can also mount the path where the gradle dependencies are stored. This lets us cache the gradle dependencies in the Github actions cache and reuse them across builds. If a developer ever uses this docker image locally, they can mount their local source code folder and the local gradle dependencies folder.

What do you think about the points I raised? Am I missing anything?

[sornian]

Ok. I may have misunderstood something. What is the purpose of the uploaded docker image? From the dockerfile in the repo, it looks like the image is used to build the Javet binaries. Is the build image primarily for people wanting to contribute to Javet development? And not meant for users of Javet?

maybe most users only use Javet products, but some customization needs should be taken into account. This requires this image to build Javet. Uploading to DockerHub is a quick way

[amithgeorge]

@caoccao An update. I was able to build the V8_Node image using a GitHub Action. It took a little over 3 hours!

The docker image (amithgeorge/javet-linux-dev:v8-10.6.194.14_node-18.10.0) contains the V8, Node and JVM. No Javet source is present, as I proposed in my earlier message.

The docker upload of the image from GitHub to DockerHub took around 7 mins.

Can you please use the image and test it? In the folder where you have the Javet source clone, please run the following. The gradle-home folder is used to cache the entire gradle home directory inside the container. I don't use gradle, so I am not sure which specific folders should we actually cache.

mkdir -p gradle-home
docker run -it --rm --name javet-dev-local \
   --volume $(pwd):/Javet --volume $(pwd)/gradle-home:/root/.gradle \
   amithgeorge/javet-linux-dev:v8-10.6.194.14_node-18.10.0 \
   bash

In the resulting shell, execute the following manually,

cd /Javet/cpp && \
  sh ./build-linux.sh -DV8_DIR=/google/v8 && \
  sh ./build-linux.sh -DNODE_DIR=/node && \
  cd ../ && \
    touch src/main/resources/libjavet-v8* && \
    gradle build test --rerun-tasks --debug && \
    touch src/main/resources/libjavet-node* && \
    gradle test --rerun-tasks --debug

exit

In your local machine, the javet jar should be present at ./build/libs/

The GitHub runner has a hardware limitations to keep in mind. I don't think we will hit them anytime soon. From https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources

Hardware specification for Windows and Linux virtual machines:
2-core CPU (x86_64)
7 GB of RAM
14 GB of SSD space

If you can confirm the docker image works, and is suitable for use, then I will continue with next steps.

---

Example workflow - https://github.com/amithgeorge/v8-node-source-binaries/blob/d71b4ea03f640587a3ed07a951fabcc3b831a218/.github/workflows/linux_build_dev.yml

Result of the action - https://github.com/amithgeorge/v8-node-source-binaries/actions/runs/3298958302

The docker image - https://hub.docker.com/layers/amithgeorge/javet-linux-dev/v8-10.6.194.14_node-18.10.0/images/sha256-42332c776174da067d0dd03dc4b97978a0c757ab44a1ac14696d53a5fad18ff7

[caoccao]

Thank @amithgeorge for the great work. I'll check that out later.

Here are some tips for your reference.

• Gradle cache cannot be easily mounted and reused. It has to be generated inside the docker build. That's why gradle assemble is called during the base image building phase. Gradle doesn't calculate universal consistent path for the cache, which means the cache path varies per user, per instance. So, I don't think your design will leverage the gradle cache.
• The current design is to minimize the time on building and testing the final binaries. Which means the work is done in the base image as much as possible.
• The Linux and Android base images share many steps till install-build-deps.sh to save time on building, pushing, pulling. Hope the new design keeps that.

[sornian]

Thank @amithgeorge for the great work。Can you upload Android and Windows image

[amithgeorge]

@sornian this is still a work in progress. The linux image is currently uploaded to my person DockerHub repo. Ideally, these images should be hosted on the Javet DockerHub. So, I would strongly advise against relying on the images I am uploading. These are mainly to test whether I am on the right track. I do not guarantee these images will remain for more than a few days.

I have never worked on Android/ARM or Windows docker images before. So I am not even sure whether I would be able to contribute towards that. My idea was to setup something for Linux, bring it under the Javet org/brand. Then @caoccao or someone with appropriate experience can rely on that to similarly implement the infra for Android and Windows.

From @caoccao's earlier message, it sounds like there is some overlap between Linux and Android images, so I might take a stab at Android as well later.

[caoccao]

I tried it out today. Thank @amithgeorge for the excellent work!

Here are few suggestions.

• It's better to call gradle assemble in the base image to prepare the gradle cache so that the actual build time will be reduced. The typical use case is Javet developers repeatedly build and test the binaries for quick verification. Keeping that loop as fast as possible should be considered seriously and the current design fulfills that.
• The actual build on top of the base image currently is inside another dockerfile. I'd like to keep this design for the simplicity. I think calling a bunch of commands is not friendly to the developers who just want to change a few lines of code to see if that works. Also, as the file systems are quite different between Windows and Linux, that dockerfile fills that gap so that building the Linux binaries on Windows reuses exactly the same one-line command.
• There are some issues #196 with the Android base image. You are welcome joining the discussion. I believe the solution would be quite different from the current approach you take for the Linux build. But, as long as they are separate images built by Github actions, I'm fine with no overlaps at all.

Please feel free to create a new branch with the changes and I'll review it whenever you want me to. Thank you again for you great help.

[josh-hemphill]

I've also been working on getting the docker builds working, and did some experimentation breaking it into a multi-stage build. I ran into issues with gclient sync since it's use of rename() fails when trying to move files across layer boundaries, and until the bug I submitted is processed, I'm doing a whole copy-delete-moveback to bring all the code into the same layer as gclient, which is kind of awful.

I'm ultimately trying to get arm64 working so it can be used on some of the raspis some of my friends are running, so I've been working with combinations of --platform= and cross-compilation flags. It seems like with the multi-stage build, I can just add a few args and have the android actually just pull from a stage of the linux build, or combine then into a single file. I'm just having trouble deciding how to do cpu architectures, since docker emulation works fine but will be a lot slower to build any non-native architectures, so I'm leaning towards custom build args to parse out what additional cross-compilers need to be installed so you can build for all your desired platforms, sharing the source code, but that's going to get very complex, so I'm not sure what the preference is.

[josh-hemphill]

There might be some confusion, I ran into the RUN gclient sync using the linux dockerfile, not the android one. It seemed to be a docker/container/overlayfs issue, nothing to do with the build target.

[caoccao]

There might be some confusion, I ran into the RUN gclient sync using the linux dockerfile, not the android one. It seemed to be a docker/container/overlayfs issue, nothing to do with the build target.

gclient sync behaves abnormally if the target_os is a non-native one, I think. That might contribute to the docker/container/overlayfs issue. I think a shell script with all these commands might work. Would you mind taking a try?

[josh-hemphill]

I don't think target_os has anything to do with the error I was having, it's never set in the linux dockerfile. I ran and observed the error in the unedited linux/base.Dockerfile with both overlayfs fuse-overlayfs and overlay2 storage divers, native and non-native filesystems, and inside and outside docker emulation; all resulted in an identical error.

[caoccao]

I don't think target_os has anything to do with the error I was having, it's never set in the linux dockerfile. I ran and observed the error in the unedited linux/base.Dockerfile with both overlayfs fuse-overlayfs and vfs storage divers, native and non-native filesystems, and inside and outside docker emulation; all resulted in an identical error.

That's weird because it works in my environment (Windows 10 + WSL 2 + Docker + Ubuntu 20.04). May I know your environment?

[josh-hemphill]

I tried on both my Ubuntu 22.04 systems, but also inside a podman container VM which uses fedora-coreos-36.20221014.2.0.

So it is the kernel that's kind of responsible, but from what I could tell, the rename() kernel function responsible hasn't changed its behavior between kernel versions, so it would have to be on some kind of wrapping library that I guess used to hide/ignore XDEV errors, but now propagates them so things know when they're breaching filesystem boundaries and loosing atomicity.

After talking to someone from the chromium mailing group, it sounds like the libs that gclient uses rely on atomicity when moving files, so this is actually better in a way since it will ensure we don't end up with gclient silently clobbering files or other undefined beaviour.

[josh-hemphill]

I found another upside to the multi-stag approach is it seems to multi-thread very nicely with the --jobs=0 option set. (though I've been setting to $(($( getconf _NPROCESSORS_ONLN ) - 1 )) so it will max out at 1 less than the number of physical cores available.)

[caoccao]

I wonder if mounting a physical volume as /google_tmp would get rid of that invalid cross-device link issue, and later copy /google_tmp to /google. Would @josh-hemphill take a try?

[josh-hemphill]

It would get rid of the error, but that's just because you're copying the data, which makes it owned by that layer. So I'm thinking a good solution is to have previous stages use another directory, then copy into the /google directory in the same layer as gclient sync and just make sure that only the /google directory gets persisted to the next stage. Whether or not to rm the old directory I think is up in the air. In my experience it does add some time to the build, but not too much. If I don't rm the old directory that stage will have a much larger footprint, but it shouldn't persist into any final image, so I'm not sure whether to do the rm or not.

[caoccao]

If I don't rm the old directory that stage will have a much larger footprint, but it shouldn't persist into any final image, so I'm not sure whether to do the rm or not.

Right, that's also one of my concerns. The image size might be nearly doubled if that's not taken well care of. Let the test tell us if that's a valid concern.