Optimize Docker image layering

[dwickern]

trafficstars

The default layering is not great for Play Framework applications. Here I create a fresh app and print the layers:

sbt new playframework/play-scala-seed.g8

// add to build.sbt
enablePlugins(DockerPlugin, LauncherJarPlugin)

[play-scala-seed] $ show dockerLayerMappings
[info] * LayeredMapping(Some(2),/Users/dwickern/code/play-scala-seed/target/scala-2.13/play-scala-seed_2.13-1.0-SNAPSHOT-sans-externalized.jar,/opt/docker/lib/com.example.play-scala-seed-1.0-SNAPSHOT-sans-externalized.jar)
[info] * LayeredMapping(Some(2),/Users/dwickern/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.16/scala-library-2.13.16.jar,/opt/docker/lib/org.scala-lang.scala-library-2.13.16.jar)
[info] * LayeredMapping(Some(2),/Users/dwickern/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/playframework/twirl/twirl-api_2.13/2.0.7/twirl-api_2.13-2.0.7.jar,/opt/docker/lib/org.playframework.twirl.twirl-api_2.13-2.0.7.jar)
[info] * LayeredMapping(Some(2),/Users/dwickern/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/playframework/play-server_2.13/3.0.6/play-server_2.13-3.0.6.jar,/opt/docker/lib/org.playframework.play-server_2.13-3.0.6.jar)
[info] * LayeredMapping(Some(2),/Users/dwickern/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/playframework/play-logback_2.13/3.0.6/play-logback_2.13-3.0.6.jar,/opt/docker/lib/org.playframework.play-logback_2.13-3.0.6.jar)
[info] * LayeredMapping(Some(2),/Users/dwickern/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/playframework/play-pekko-http-server_2.13/3.0.6/play-pekko-http-server_2.13-3.0.6.jar,/opt/docker/lib/org.playframework.play-pekko-http-server_2.13-3.0.6.jar)
[info] * LayeredMapping(Some(2),/Users/dwickern/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/playframework/play-filters-helpers_2.13/3.0.6/play-filters-helpers_2.13-3.0.6.jar,/opt/docker/lib/org.playframework.play-filters-helpers_2.13-3.0.6.jar)
...
[info] * LayeredMapping(Some(2),/Users/dwickern/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/google/guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar,/opt/docker/lib/com.google.guava.listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar)
[info] * LayeredMapping(Some(2),/Users/dwickern/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar,/opt/docker/lib/com.google.code.findbugs.jsr305-3.0.2.jar)
[info] * LayeredMapping(Some(2),/Users/dwickern/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/checkerframework/checker-qual/3.37.0/checker-qual-3.37.0.jar,/opt/docker/lib/org.checkerframework.checker-qual-3.37.0.jar)
[info] * LayeredMapping(Some(2),/Users/dwickern/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/google/j2objc/j2objc-annotations/2.8/j2objc-annotations-2.8.jar,/opt/docker/lib/com.google.j2objc.j2objc-annotations-2.8.jar)
[info] * LayeredMapping(Some(2),/Users/dwickern/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/pekko/pekko-protobuf-v3_2.13/1.0.3/pekko-protobuf-v3_2.13-1.0.3.jar,/opt/docker/lib/org.apache.pekko.pekko-protobuf-v3_2.13-1.0.3.jar)
[info] * LayeredMapping(Some(2),/Users/dwickern/code/play-scala-seed/target/scala-2.13/play-scala-seed_2.13-1.0-SNAPSHOT-web-assets.jar,/opt/docker/lib/com.example.play-scala-seed-1.0-SNAPSHOT-assets.jar)
[info] * LayeredMapping(Some(2),/Users/dwickern/code/play-scala-seed/target/scala-2.13/com.example.play-scala-seed-1.0-SNAPSHOT-launcher.jar,/opt/docker/lib/com.example.play-scala-seed-1.0-SNAPSHOT-launcher.jar)
[info] * LayeredMapping(Some(4),/Users/dwickern/code/play-scala-seed/target/universal/scripts/bin/play-scala-seed,/opt/docker/bin/play-scala-seed)
[info] * LayeredMapping(Some(4),/Users/dwickern/code/play-scala-seed/target/universal/scripts/bin/play-scala-seed.bat,/opt/docker/bin/play-scala-seed.bat)
[info] * LayeredMapping(Some(1),/Users/dwickern/code/play-scala-seed/conf/logback.xml,/opt/docker/conf/logback.xml)
[info] * LayeredMapping(Some(1),/Users/dwickern/code/play-scala-seed/conf/messages,/opt/docker/conf/messages)
[info] * LayeredMapping(Some(1),/Users/dwickern/code/play-scala-seed/conf/application.conf,/opt/docker/conf/application.conf)
[info] * LayeredMapping(Some(1),/Users/dwickern/code/play-scala-seed/conf/routes,/opt/docker/conf/routes)
[info] * LayeredMapping(None,/Users/dwickern/code/play-scala-seed/target/scala-2.13/api,/opt/docker/share/doc/api/)
[info] * LayeredMapping(None,/Users/dwickern/code/play-scala-seed/target/scala-2.13/api/index.html,/opt/docker/share/doc/api//index.html)
[info] * LayeredMapping(None,/Users/dwickern/code/play-scala-seed/target/scala-2.13/api/index.js,/opt/docker/share/doc/api//index.js)
[info] * LayeredMapping(None,/Users/dwickern/code/play-scala-seed/target/scala-2.13/api/lib,/opt/docker/share/doc/api//lib)
[info] * LayeredMapping(None,/Users/dwickern/code/play-scala-seed/target/scala-2.13/api/lib/source-code-pro-v6-latin-regular.ttf,/opt/docker/share/doc/api//lib/source-code-pro-v6-latin-regular.ttf)
[info] * LayeredMapping(None,/Users/dwickern/code/play-scala-seed/target/scala-2.13/api/lib/annotation_comp.svg,/opt/docker/share/doc/api//lib/annotation_comp.svg)
[info] * LayeredMapping(None,/Users/dwickern/code/play-scala-seed/target/scala-2.13/api/lib/abstract_type.svg,/opt/docker/share/doc/api//lib/abstract_type.svg)
...
[info] * LayeredMapping(None,/Users/dwickern/code/play-scala-seed/target/scala-2.13/api/router/Routes.html,/opt/docker/share/doc/api//router/Routes.html)
[info] * LayeredMapping(None,/Users/dwickern/code/play-scala-seed/target/scala-2.13/api/router/index.html,/opt/docker/share/doc/api//router/index.html)
[info] * LayeredMapping(None,/Users/dwickern/code/play-scala-seed/target/scala-2.13/api/router/RoutesPrefix$.html,/opt/docker/share/doc/api//router/RoutesPrefix$.html)

The biggest issue is that application artifacts are put in the same layer as libraryDependencies. The library dependencies are 43MB for this play-scala-seed app and much larger for any real application. Dependencies change less frequently compared to application code so we should be able to cache them across builds.

Current layers

Here's how the layers are currently structured:

Layer 1

• sbt-native-packager's generated conf/application.ini if using Universal / javaOptions
• Play Framework's externalized resources (by default, the contents of conf/)

Layer 2

• the project's transitive libraryDependencies
• sbt-native-packager's launcher jar if using LauncherJarPlugin
• Play Framework's -assets.jar and -sans-externalized.jar jars

Layer 3

• sbt-native-packager jlink files if using JlinkPlugin

Layer 4

• the project's transitive artifact jars
• sbt-native-packager's shell scripts
• sbt-native-packager's classpath jar if using ClasspathJarPlugin

Final layer (layerId = None)

• sbt-native-packager's mappings of Docker / sourceDirectory
• Play Framework's API docs if using includeDocumentationInBinary := true (default true)

Proposed layers

At minimum we should move the project artifacts out from layer 2. Config files shouldn't be the bottom layer. They are only a few KB so it doesn't matter to cache them. Should jlink files be layered below libraryDependencies? I would guess they change less frequently but I don't use jlink 🤷

[muuki88]

Hi @dwickern

That's indeed very wasteful. I remember vaguely that this was solved at one point, but apparently not 😢

I assume this can be fixed by re-ordering the dockerCommands ? https://github.com/sbt/sbt-native-packager/blob/03f8a0e054266ba438e0798c9c990bd3210eb85a/src/main/scala/com/typesafe/sbt/packager/docker/DockerPlugin.scala#L188-L271

[dwickern]

It's actually dockerGroupLayers. it will be easy to fix... more work to update the tests 😅 https://github.com/sbt/sbt-native-packager/blob/03f8a0e054266ba438e0798c9c990bd3210eb85a/src/main/scala/com/typesafe/sbt/packager/docker/DockerPlugin.scala#L108-L140

#1425 made some improvements before. The layering is pretty good for most applications. The issue is Play Framework outputs additional jars besides the default packageBin