Updates to existing tags break deployments

[aschlei]

We deploy a testing environment as a stack of docker services multiple times a day. On Monday afternoon (Feb 22), this deployment broke due to the Java upgrade from 8 to 11 made in the image. We are using the 2.2.0 image. When this latest update went through, the 2.2.0 tag was updated and there is no tag pointing to the previous (working) version. This has actually happened a few times over the last couple years.

Currently I am seeing the following errors in our worker logs. It's not clear to me where these originate from right off hand. We'll have to investigate. But in the meantime, our deployment remains broken. If the previous tag was available, we could get it working again.

Unrecognized VM option 'PrintGCDateStamps'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Expected behavior

When updates are made to this image, the existing images should still be accessible. One way to do this would be to have tags like 2.2.0-1, 2.2.0-2, 2.2.0-3, etc. The 2.2.0 tag could still exist and point to the latest 2.2.0 version. This would allow users to point to 2.2.0-1 for example and not have to worry about an update breaking things; they can upgrade to a more recent tag after testing the changes to verify things still work or making the appropriate modifications.

Another option might be to maintain separate images/tags for major Java versions.

It would also be nice if the previous version (with Java 8) could be made available on Docker Hub again.

Actual behavior

When a change is pushed, the existing tags are overwritten and there's no way to get back to a previous working version.

Steps to reproduce the behavior

Have a Docker image that is built FROM storm:2.2.0 or a compose file that references storm:2.2.0. When that tag is replaced, this may break something that was previously working.

[compmodder26]

I too am facing this issue. I get the same error as posted above.

Unrecognized VM option 'PrintGCDateStamps' Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit.

I thought perhaps it might be because my topology code was compiled against java 8, but compiling with java 11 doesn't change the result.

[31z4]

Hi @aschlei and @compmodder26, thanks for reporting this! I apologise that Java upgrade broke your deployments. I took a note and will make sure that existing tags won't be overwritten on potentially breaking changes.

[chriskresge]

I had this issue also. Adding an updated "worker.childopts" in storm.yaml fixed it for me:

worker.childopts: "-Xmx1574m -Xlog:gc:artifacts/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump"

[rzo1]

Yup. The default worker.childopts do not work with Java 11, if no custom config is given to the container. I solved it by overriding the default config, which is generated via docker-entrypoint.sh.

@31z4 Do you generally accept PR? If so, I could probably submit my changes to the entrypoint script.

[xcoder123]

@rzo1 Can you share more details how you fixed it?

[rzo1]

Yes. In the end, I ended up overriding the related docker-entrypoint.sh and building my custom container (custom provided libs, log config) on the basis of this container, ie I added some custom entries

supervisor.childopts: "-Xmx2G"
worker.childopts: "--add-opens java.base/jdk.internal.loader=ALL-UNNAMED -Xmx2G"

to docker-entrypoint.sh` overriding the defaults. Hope this helps...

[xcoder123]

Yup your approach worked. And while doing it I found another approach that works

After I did the Dockerfile modification, I realised I'm changing the config to all systems, nimbus, supervisor, ui etc.

Before I tried adding storm.yaml config file with "worker.childopts" to just nimbus. If you add it to all, then you don't need to rebuild the original image.