modelmesh-serving TODO: Create architectural decision record issue to summarize why we want to use `latest` images:

TODO: Create architectural decision record issue to summarize why we want to use `latest` images:

Open ckadner opened this issue 1 year ago • 4 comments

TODO: Create architectural decision record issue to summarize why we want to use latest images:

Cons of using latest:

MM image builds might fail unexpectedly
hard to debug/trace back when failing changes happened
last working image may not be possible to recreate for a new MM release without any changes
last working image might not be available even if identified
even if builds succeed, there might be unexpected changes in behavior which happens quite often with 3rd party dependencies of Python packages (on a tangent: when we update to Go 1.19 a while ago, the go get CLI tools install commands "succeed" but no longer installed executables, though we would not use ubi8/go-toolset:latest)

Pros of using latest:

keeping specific version tag, we keep outdated code, old vulnerabilities
we don't have to keep updating to newer patched versions of base images when upstream base images get updated frequently

Builder vs final images:

using latest on a final build stage (not builder image) should be fairly "safe", no 3rd party libraries are installed, just files copied from builder image.
however, it's the base of the builder images where most problems/vulnerabilities are coming in/need to be fixed, are getting fixed upstream,
so really, all build stages need to build on a latest upstream (UBI) base image

Finally, we should follow the same strategy for all MM images consistently.

Originally posted by @ckadner in https://github.com/kserve/rest-proxy/issues/43#issuecomment-2138520449

May 30 '24 01:05 ckadner

FYI @spolti @Jooho @njhill @tjohnson31415 @rafvasq

May 30 '24 01:05 ckadner

We could add a nightly build to keep updated about the build status -- to not be surprised by failing builds when we start working on a release.

Additionally, failing builds could trigger notifications to be send to the "modelmesh-build" team. Team to be created.

Have a build matrix even to build several versions of UBI 8/9

Jun 04 '24 16:06 ckadner

@spolti with respect to FIPS we stay with UBI 8 for the time being. All the remaining MM images use UBI 8 latest for final build stage. Thanks!

Jun 11 '24 16:06 ckadner

About the FIPS, there is no public document yet, but the work still on going.

Jun 12 '24 15:06 spolti