Make it easier to package certain content in the root of a fat jar

[wilkinsona]

Spring Boot 1.4 has made using this technique for including a Java agent in a fat jar more difficult. The problem is that the agent's classes get repackaged into BOOT-INF/classes but they need to stay in the root of the jar.

You can work around it by using a separate module to consume the fat jar and add the agent to it but it'd be nice if users didn't have to jump through that extra hoop. One solution would be to add something to the build plugins that allow a user to mark certain classes as having to stay in the root of the jar.

[10168852]

in spring boot 1.4 we could have a deployable be a dependency to another project. Now it can't cause being a dependency maven isn't able to find the classes because they are located at BOOT-INF/classes

[snicoll]

@10168852 this is unrelated to this issue and I frankly consider this to be an improvement. Using a fat jar as a dependency means that you take the complete dependency tree with it. That deployable of yours must be quite large. Please have a look to this stackoverflow thread.

[wilkinsona]

#2268 is somewhat related to this

[reachym]

@wilkinsona We are facing the issue you mentioned above as we currently use the fat jar for Newrelic's javaagent. We recently upgraded to 1.4 version of Spring Boot and due to the different structure, it's not working anymore. Could you please elaborate a bit more on "You can work around it by using a separate module to consume the fat jar and add the agent to it"? I know it's probably unrelated here, but it would be great if you can provide the alternative while you are discussing the issue. Thanks!

[wilkinsona]

Here's an example with Gradle that doesn't require a separate module. The jarWithAgent task will produce a second jar that contains New Relic's agent alongside the fat jar:

buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath 'org.springframework.boot:spring-boot-gradle-plugin:1.4.1.RELEASE'
    }
}

repositories {
    mavenCentral()
}

apply plugin: 'spring-boot'

springBoot {
    mainClass 'com.example.Main'
}

configurations {
    newrelic
}

dependencies {
    compile 'org.springframework.boot:spring-boot-starter-web'
    newrelic 'com.newrelic.agent.java:newrelic-agent:3.12.1'
}

task extractManifest(type: Copy) {
    dependsOn bootRepackage
    from(zipTree(jar.outputs.files.singleFile))
    include 'META-INF/MANIFEST.MF'
    into 'build/extracted'
}

task jarWithAgent(type: Jar) {
    dependsOn extractManifest
    classifier 'with-agent'
    entryCompression ZipEntryCompression.STORED
    from(
        zipTree(jar.outputs.files.singleFile),
        zipTree(configurations.newrelic.singleFile)
    )
    manifest {
        from new File(extractManifest.outputs.files.singleFile, '/META-INF/MANIFEST.MF')
        attributes([
            'Premain-Class': 'com.newrelic.bootstrap.BootstrapAgent',
            'Can-Redefine-Classes': 'true',
            'Can-Retransform-Classes': 'true'
        ])
    }
}

[marcosbarbero]

@wilkinsona is there at tip to achieve same thing using maven?

[wilkinsona]

Another use case for this is Cloud Foundry's pre-runtime hooks that need to be placed in the root of an application's directory. When you're pushing a fat jar, that means they need to go in the root of the archive. That's trickier in 1.4 as anything that was in src/main/resources will now be repackaged into BOOT-INF/classes.

[wilkinsona]

Here's an example for the New Relic agent based on a prototype for this issue:

configurations {
    newrelic
}

dependencies {
    compile 'org.springframework.boot:spring-boot-starter-web'
    newrelic 'com.newrelic.agent.java:newrelic-agent:3.12.1'
}

jar {
    // Include the contents of the New Relic jar
    from(zipTree(configurations.newrelic.singleFile))
    // Set the manifest attributes that the agent needs
    manifest {
        attributes([
            'Premain-Class': 'com.newrelic.bootstrap.BootstrapAgent',
            'Can-Redefine-Classes': 'true',
            'Can-Retransform-Classes': 'true'
        ])
    }
}

bootRepackage {
    // rootEntries are those that should not be repackaged. They're set here by
    // collecting the names of all the entries in the New Relic jar. They are matched
    // using startsWith, for example "com/newrelic" would keep "com/newrelic" and
    // anything beneath it in the root of the jar. Using the names of all the entries
    // means the configuration is more concise.
    rootEntries = new java.util.jar.JarFile(configurations.newrelic.singleFile).entries().collect { it.name }
}

[wilkinsona]

Upon closer inspection, I'm confused about the Cloud Foundry side of this.

@kelapure My understanding is that you're pushing a jar with a .profile.d directory that contains one or more scripts. I guess you're then relying on the fact that the Java build pack unpacks a jar before running the app so that you end up with a .profile.d directory in the root of the app's directory. Correct so far?

My confusion arises because the Cloud Foundry documentation says

The Java buildpack does not support pre-runtime hooks.

And

Your app root directory may also include a .profile.d directory that contains bash scripts that perform initialization tasks for the buildpack. Developers should not edit these scripts unless they are using a custom build pack

So either files in .profile.d are pre-runtime hooks and aren't supported by the Java buildpack, or they're something else and should be used in conjunction with a custom build pack. If you're using a custom build pack then you can easily move things around to suit your needs without a change in Boot.

@nebhale Can you clarify things from a CF build pack perspective please?

[nebhale]

A more accurate way of writing "does not support" is that their usage is undefined when using the Java Buildpack. Because these hooks have to ability to change the environment of a running application, they can also invalidate assumptions that the buildpack makes about the environment. Therefore, if you choose to use them, any strange behavior in the buildpack or at runtime is your responsibility. The actual functionality of executing these hooks is owned by the container, not the buildpacks though, so they will get executed no matter what.

That being said, we publicly and strongly discourage the use of these hooks as they encourage including environment-specific functionality within an application, violating one of the major tenants of 12-Factor applications. For example, using the .profile support means that your application may no longer runs the same way locally during development as it does inside of the CF container.

The Java Buildpack views and encourages integrations such as these to be orthogonal and provides extension points to facilitate that behavior. In fact, New Relic was the very first orthogonal integration we did (it works for any JVM-based application, with no application-specific configuration) and we used it to prove out this idea and design.

[kelapure]

My specific use-case for a .profile.d directory was for debugging non-heap OOMs. We worked around the current issue, by coding a custom spring boot actuator metric library to record native memory statistics resulting from -XX:NativeMemoryTracking=summary. see https://github.com/mcabaj/nmt-metrics

From a cloud foundry perspective my view is that runtime hooks should only be relied on specific instances of debugging and troubleshooting and should not be the norm.

There are several instances of older frameworks/jars/apps that package content at the root of the jar. Boot should have a provision 1. to allow the packaging of these resources at the root say a .profile.d directory and 2. configure boot to add them to the classpath.

I agree this issue is an enhancement not a trivial one though since .profile.d is just one of the use cases of packaging resources at the top of the jar.

[wilkinsona]

Thanks, @nebhale and @kelapure.

From a Cloud Foundry perspective, I'm specifically interested in the .profile.d.

1. to allow the packaging of these resources at the root say a .profile.d directory

My concern about packaging the .profile.d directory at the root of the jar is that you're relying on the Java Buildpack unpacking the jar before its run so that the .profile.d directory ends up in the root of the application's directory. I'm not sure if that's an internal detail of how the buildpack works, or something that can be safely relied upon. What's your take on that, @nebhale?

1. configure boot to add them to the classpath.

Anything packaged in the root of the jar is automatically on the class path. Specifically, it's on the class path of the system class loader which is the parent of Boot's class loader that's created by the launcher. Note that this means that the technique that's currently being used with the .profile.d directory results in that directory being on the application's class path when it doesn't need to be. That contributes to my concern about the technique and does make it feel like something of a hack.

In summary, there are two separate use cases here:

1. Packaging something like a Java agent or custom FileSystem at the root of the jar so that it's loaded by the system class loader. Thanks to the Java buildpack's specific support for things like New Relic, this use case is largely applicable outside of Cloud Foundry.
2. Controlling the contents of a .profile.d directory in the root of the application's directory when deployed to Cloud Foundry.

Both use cases can be satisfied by the same solution, however exactly how we might name and document that solution may differ quite a lot depending on which use case the user is interested in. The first use case is sound and the solution can easily be described in terms of marking things that should be visible to the system class loader. The second use case is less sound as it feels like it's using the jar as something of a trojan horse to smuggle the .profile.d directory into the root of the application's directory after it's been unpacked.

[nebhale]

@wilkinsona Actually, the buildpack doesn't unpack archives, Cloud Foundry does it. The contract for staging an application is that the buildpack is presented with an exploded archive no matter what. So if you push a folder full of Ruby application, the CLI zips it, sends it over and the server unzips that in a container and presents it to the buildpacks. If you push a JAR file, the CLI just sends it as-is (it actually removes some pieces that it already knows about, but that's an optimization) and the server unzips that in a container and presents it to the buildpacks.

My main problem with this whole thing is that the design of the .profile and .profile.d feature in Cloud Foundry doesn't take into account how Java classpaths work. As you say, putting anything in the root of the application (which is what is required by this feature) means that it will be exposed on the class path of the application, which can lead to unanticipated behavior.

[wilkinsona]

Thanks, @nebhale.

So, to summarise:

• The fact that the archive is unzipped can be relied upon
• Putting stuff in the root of the jar to get it into the root of the application directory is a hack that we don't recommend

That convinces me that this enhancement should be described solely in terms of packaging entries in the jar such that they are on the class path of the system class loader.

[dsyer]

Another twist on this is that META-INF/aop.xml is packaged in the root of the fat jar by default, but actually that causes some problems at runtime, and it might be better to put it in BOOT-INF/classes instead (ref #7587).

[peterabbott]

I see this is still open for discussion and wondering if anything is going to happen???

I have been searching the forums for a few days as I am keen to upgrade from 1.3.x. The case we use is that we bundle the agent jar with our application as there is a dependency between the two (in our case the Jetty Http2 libs). Tried many different gradle hacks from updating the repackaged jar to defining custom layouts, all seem a bit overkill and brittle.

Starting to think that updating the provisioning process to pull in the agent is an easier option, but would have been nice to have the option to include a "custom system classpath" on the repackaged jar.

[youngm]

@wilkinsona Sure, using .profile can be considered a hack. But sometimes a hack is the appropriate solution to a problem for an end user application.

The java-buildpack doesn't today actively delete .profile files added to user artifacts. That would be ridiculous. But by not supporting this at the build layer you are essentially doing just that to users who need a .profile hack in their solution.

That said I don't really care about the semantics as long as support for putting stuff in the system class loader will allow for a .profile hack.

[gebhardt]

@marcosbarbero Here is a Maven example how you can extract and add a Jar to the Spring Boot executable jar:

        <plugin>
            <artifactId>maven-antrun-plugin</artifactId>
            <executions>
                <execution>
                    <id>addExtractedJarOnRootLevel</id>
                    <phase>package</phase>
                    <configuration>
                        <target>
                            <zip destfile="${project.build.directory}/${project.artifactId}-${project.version}-exec.jar"
                                update="yes" compress="false">
                                <zipfileset src="${your-groupId:your-arctifactId:jar}"/>
                            </zip>
                        </target>
                    </configuration>
                    <goals>
                        <goal>run</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>

Please note that you can directly reference your Maven dependency in the Ant Skript by replacing your-groupIdand your-arctifactId accrodingly.

[marcosbarbero]

@gebhardt thanks a lot :) @RichardCSantana @matheusgg take a look on previous comment.

[timomeinen]

@gebhardt Note: Ant is only able to update the JAR, if it is not build as executable. Otherwise the embeddedLaunchScript is put in front of the JAR and cannot be processed by zip or jar tool.

[harezmi]

Support to exclude some of those resources moving into BOOT-INF/classes is also needed when someone aims to place JSP files under root jar path /META-INF/resources (According to JSP spec, you are allowed to place JSPs under that folder in your jar file). Currently files placed within src/main/resources/META-INF/resources are all moved into BOOT-INF/classes which causes embedded tomcat to fail those JSP files loaded.

[kelapure]

This is a very common use case in replatforming legacy apps where we need to package files outside the app itself for instance when the app relies on the location of a certain file in the container.

[wilkinsona]

Right now, for those for whom this is important, I would recommend using Gradle rather than Maven to build your application. Alternatively, you can achieve this in a few different ways using Maven as shown in the comments above such as this one from @gabhardt.

when the app relies on the location of a certain file in the container

As described by @nebhale above, neither the Boot team nor the CF Java Buildpack team recommend using this mechanism to achieve that:

My main problem with this whole thing is that the design of the .profile and .profile.d feature in Cloud Foundry doesn't take into account how Java classpaths work. As you say, putting anything in the root of the application (which is what is required by this feature) means that it will be exposed on the class path of the application, which can lead to unanticipated behavior.

This reasoning extends beyond .profile and .profile.d to any file that's bundled in the root of the archive.

[andy722]

In my case (bundling custom Charset implementations wired via SPI in a boot jar), there's a simple solution:

configurations {
    charsets
}

dependencies {
    // ...
    charsets group: "...", name: "...", version: "...", ext: "jar"
}

bootJar {
    from (
        zipTree(configurations.charsets.singleFile)
    )
}

Hope this helps someone)

[dsyer]

Bumping this because now Cloud Native Buildpacks are in the picture and Spring Boot can build OCI images, which it couldn't back when the issue was first raised. Putting a Procfile at the root of a jar is explicitly supported by the Paketo buildpacks. It's not a common use case, but sometimes you just don't have any other way of tweaking the app to do what you want it to. In my case I wanted to add a new entrypoint to the image. The antrun-plugin hack for Maven (or the equivalent for Gradle) won't work in this case because Spring Boot doesn't unpack the JAR file over to the buildpack, it just copies the files it thinks are there.

[wilkinsona]

The antrun-plugin hack for Maven (or the equivalent for Gradle) won't work in this case because Spring Boot doesn't unpack the JAR file over to the buildpack, it just copies the files it thinks are there.

It will work with Gradle as it uses the output of the bootJar task as an input into the bootBuildImage task. Something modelled on the following should be all it takes with Gradle:

bootJar {
    from (
        // …
    )
}

It may be impossible with Maven if you're using CNBs at the moment so perhaps this issue needs to be retitled. We wondered a few times if we should rework the Maven CNB support to use the output of repackage as Gradle uses the output of bootJar. I'll flag this for team discussion so we can go round that loop again.

[scottfrederick]

For discussion, there is a proposal to address this with an enhancement to the Paketo CNB buildpacks: https://github.com/paketo-buildpacks/procfile/issues/45.

[wilkinsona]

In light of the build packs enhancement, no changes are need in Boot for Procfile support.

[linux-china]

Any clue to include Procfile in Fatjar for buildpack? Now I use following commands to build Docker image with buildpack.

mvn -DskipTests clean package
jar uvf target/spring-boot25-demo-0.0.1-SNAPSHOT.jar Procfile
pack build --builder paketobuildpacks/builder:full --path target/spring-boot25-demo-0.0.1-SNAPSHOT.jar spring-boot25-demo:0.0.1

[wilkinsona]

@linux-china The recommended approach is to use a binding to provide the Procfile. See the README for a bit more info.