Add pack option to the builder options for cloud native buildpacks

[nickhammond]

This PR introduces Cloud Native Buildpacks to the list of builder options for Kamal.

This opens up the option to utilize buildpacks instead of writing a Dockerfile from scratch for each app that you want to deploy with Kamal. The end result is still an OCI-compliant docker image that you can run the same as the currently built docker images with Kamal.

You can also use any buildpacks or builders that you'd like so if you prefer some of the Paketo buildpacks instead you can use those too. The example below is utilizing Heroku's builder with the ruby and procfile buildpack which gives you the familiar Heroku build process when you deploy your application. Auto-detection of bundler, cache management of gem and asset installation, and various other features that come along with those.

With this PR you'd need to have pack installed as well as Docker and then within your deploy.yml change your builder to:

builder:
  arch: amd64
  pack:
    builder: heroku/builder:24
    buildpacks:
    - heroku/ruby
    - heroku/procfile

The default process that the buildpack tries to boot is the web process, you can add a Procfile for this:

web: ./bin/docker-entrypoint ./bin/rails server

And lastly, buildpacks don't bind to a default port so you'll either need to set proxy.app_port(Kamal 2.0 / kamal-proxy) to your application port or set your app to use port 80 which is the default kamal-proxy port.

Option 1 (Assuming your app uses port 3000):

proxy:
  app_port: 3000

Option 2 (Rails app running Puma that supports setting $PORT):

env:
  clear:
    PORT: 80

servers:
  web:
    hosts:
      - 123.456.78.9

Buildpacks work in a detect and then build flow. The detect step looks for common files or checks that indicate it is indeed a ruby application by looking for a Gemfile.lock for instance. If the detect step passes then it triggers the build phase which essentially triggers a bundle install in this example.

With heroku/builder:24 so far I've found that the image size is about the same, it's only 2mb off for a 235mb image. Build time is typically faster with pack but depends on how well you've optimized your Dockerfile. The win though is not having to think about how to cache your gem installs, node installs or any other package manager installs that have a buildpack. It's also following the common conventions for building containers and various stumbling blocks that Heroku and others have been blazing through over the years.

Kamal discussion: https://github.com/basecamp/kamal/discussions/795 Heroku discussion: https://github.com/heroku/buildpacks/discussions/6 Heroku official buildpacks: https://devcenter.heroku.com/articles/buildpacks Heroku 3rd party buildpacks: https://elements.heroku.com/buildpacks Full setup overview: https://www.fromthekeyboard.com/deploying-a-rails-app-with-kamal-heroku-style/

Todos:

• [x] Companion PR for kamal-site Started in https://github.com/basecamp/kamal-site/pull/117
• [x] Excluded files mention via project.toml
• [x] Catch up with main
• [x] ~~Discuss potential for a remote pack option~~ Out of scope
• [x] Test pack with the git archive context
• [x] What should kamal build create do when using buildpacks? Just point to the install docs? https://buildpacks.io/docs/for-platform-operators/how-to/integrate-ci/pack/ - Since you don't typically call kamal build create and it's instead called within a build I'm going to close this one out.
• [x] Update kamal build details to run pack version && pack builder inspect
• [x] ~~Does kamal build remove need to do anything?~~ Since we're not creating a build context like you normally would with Docker there's nothing to actually remove. Is there anything in the Kamal lifecycle though that we at least need a no-op method for this?

[alexohre]

hey @nickhammond and @dhh, does this change resolve the custom build issue like using the builder of choice. e.g. docker build cloud?

[nickhammond]

Hey @alexohre - No, that'll just be a remote builder with the engine pointing to Docker cloud(https://github.com/basecamp/kamal/discussions/914) which would be a different PR. The builders were just reorganized a bit as well so it might be simpler to add an additional option for a remote cloud builder in a different PR.

[alexohre]

Hey @alexohre - No, that'll just be a remote builder with the engine pointing to Docker cloud(https://github.com/basecamp/kamal/discussions/914) which would be a different PR. The builders were just reorganized a bit as well so it might be simpler to add an additional option for a remote cloud builder in a different PR.

Oh, thanks for the awareness. I would be glad if you could help me make a PR for that since I don't know how to build gems or modify it for now

[dhh]

This is fascinating work, @nickhammond. I'm surprised by how unobtrusive it is! But I'd like to understand the whole flow better. I'm not sure this is going to be all that relevant for Rails apps that now already come with well-optimized Dockerfiles out of the box, but I could see how that may well be different if you're doing a Sinatra app or some app from another framework that doesn't provide that.

Could you show how the entire flow would go with, say, a Sinatra app, using buildpacks, and deploying on something like Digital Ocean? Want to make sure that this isn't tied to any one company or platform.

[nickhammond]

Hey @dhh, thanks for taking a look!

I think adding support for buildpacks will be great for the adoption of Kamal but you can always still reach for the sharper tool(a full Dockerfile) when needed.

I built out a few hello world examples, the main thing is just making sure your app boots on port 80 for kamal-proxy or just ensuring that you set your app_port if it doesn't. This isn't a buildpack-specific thing but more of a change that came with kamal-proxy, packs don't export a port by default.

Here are the hello world apps that I built and tested out on Digital Ocean and wrote a more detailed overview for the whole process as well.

• kamal-buildpacks-node
• kamal-buildpacks-php
• kamal-buildpacks-rails - Single-file Rails app
• kamal-buildpacks-sinatra

[hone]

@nickhammond thanks for all your investigations and opening this PR. :heart:

Want to make sure that this isn't tied to any one company or platform.

@dhh :wave: It's been a while. As Cloud Native Buildpacks (CNB) maintainer I'm biased and would love to see this supported in kamal. :)

Nick touches on this in his blog, but if it's any assurance CNB as an upstream project is a CNCF Incubation project which pushes for not being a single vendor OSS project. In fact, the project was started from the get go by two companies, Heroku & Pivotal. It's really about bringing that Heroku magic to container image building, transforming your app source code into an OCI image (No Dockerfile needed). You can push image to a registry, docker run it locally, or even use it as a base image in the FROM directive of a Dockerfile. If you don't want to use the Heroku builder and buildpacks, there`s the Paketo ones, and you can also write your own.

[nickhammond]

Started on the docs in this kamal-site PR https://github.com/basecamp/kamal-site/pull/117.

[nickhammond]

@nickhammond - I noticed in your sample apps, that you've set the context for the builder to ., which avoids using the git clone for building.

Is that just a preference or is there any reason it would be required?

@djmb I usually just use the old context style when initially getting a project going since it's a bit faster. I've removed the context on all of the sample apps and they're all deploying successfully.

To be honest though I'm not super familiar with how the clone process actually works. I'm passing in the same builder_context to the pack CLI that's currently being done with the Docker build and for both I'm seeing the context as "." in the output.

Full pack command that's running:

  INFO [d6b21e93] Running /usr/bin/env pack build nickhammond/hotdonuts --platform linux/amd64 --builder heroku/builder:24 --buildpack heroku/ruby --buildpack heroku/procfile --buildpack paketo-buildpacks/image-labels -t nickhammond/hotdonuts:eca189d62a8c2ba97fdfca5f85699a50a6d50ce4 -t nickhammond/hotdonuts:latest --env BP_IMAGE_LABELS=service=hotdonuts --path . && docker push nickhammond/hotdonuts:eca189d62a8c2ba97fdfca5f85699a50a6d50ce4 && docker push nickhammond/hotdonuts:latest as n@localhost

Vs. Docker command:

  INFO [e3db1e68] Running docker buildx build --push --platform linux/amd64 --builder kamal-local-docker-container -t nickhammond/hotdonuts:eca189d62a8c2ba97fdfca5f85699a50a6d50ce4 -t nickhammond/hotdonuts:latest --label service="hotdonuts" --file Dockerfile . as n@localhost

With both though I'm seeing the clone steps, does it clone into that temp directory and then drop into it?

  INFO Cloning repo into build directory `/var/folders/6q/53gfp0q92gngndncp5mmk9cr0000gn/T/kamal-clones/hotdonuts-39f3a8537243e/hotdonuts-sinatra/`...
  INFO [a351ef89] Running /usr/bin/env git -C /var/folders/6q/53gfp0q92gngndncp5mmk9cr0000gn/T/kamal-clones/hotdonuts-39f3a8537243e clone /Users/n/src/hotdonuts-sinatra --recurse-submodules as n@localhost
  INFO Resetting local clone as `/var/folders/6q/53gfp0q92gngndncp5mmk9cr0000gn/T/kamal-clones/hotdonuts-39f3a8537243e/hotdonuts-sinatra/` already exists...
  INFO [a0986137] Running /usr/bin/env git -C /var/folders/6q/53gfp0q92gngndncp5mmk9cr0000gn/T/kamal-clones/hotdonuts-39f3a8537243e/hotdonuts-sinatra/ remote set-url origin /Users/n/src/hotdonuts-sinatra as n@localhost
  INFO [a0986137] Finished in 0.005 seconds with exit status 0 (successful).
  INFO [13044d07] Running /usr/bin/env git -C /var/folders/6q/53gfp0q92gngndncp5mmk9cr0000gn/T/kamal-clones/hotdonuts-39f3a8537243e/hotdonuts-sinatra/ fetch origin as n@localhost
  INFO [13044d07] Finished in 0.019 seconds with exit status 0 (successful).
  INFO [05383764] Running /usr/bin/env git -C /var/folders/6q/53gfp0q92gngndncp5mmk9cr0000gn/T/kamal-clones/hotdonuts-39f3a8537243e/hotdonuts-sinatra/ reset --hard eca189d62a8c2ba97fdfca5f85699a50a6d50ce4 as n@localhost
  INFO [05383764] Finished in 0.007 seconds with exit status 0 (successful).
  INFO [97641014] Running /usr/bin/env git -C /var/folders/6q/53gfp0q92gngndncp5mmk9cr0000gn/T/kamal-clones/hotdonuts-39f3a8537243e/hotdonuts-sinatra/ clean -fdx as n@localhost
  INFO [97641014] Finished in 0.006 seconds with exit status 0 (successful).
  INFO [8b0ae09b] Running /usr/bin/env git -C /var/folders/6q/53gfp0q92gngndncp5mmk9cr0000gn/T/kamal-clones/hotdonuts-39f3a8537243e/hotdonuts-sinatra/ submodule update --init as n@localhost
  INFO [8b0ae09b] Finished in 0.047 seconds with exit status 0 (successful).
  INFO [753722cb] Running /usr/bin/env git -C /var/folders/6q/53gfp0q92gngndncp5mmk9cr0000gn/T/kamal-clones/hotdonuts-39f3a8537243e/hotdonuts-sinatra/ status --porcelain as n@localhost
  INFO [753722cb] Finished in 0.006 seconds with exit status 0 (successful).
  INFO [0e2a42fa] Running /usr/bin/env git -C /var/folders/6q/53gfp0q92gngndncp5mmk9cr0000gn/T/kamal-clones/hotdonuts-39f3a8537243e/hotdonuts-sinatra/ rev-parse HEAD as n@localhost
  INFO [0e2a42fa] Finished in 0.005 seconds with exit status 0 (successful).
  INFO [7ba57d96] Running /usr/bin/env  as n@localhost
  INFO [7ba57d96] Finished in 0.002 seconds with exit status 0 (successful).
  INFO [d6b21e93] Running /usr/bin/env pack build nickhammond/hotdonuts --platform linux/amd64 --builder heroku/builder:24 --buildpack heroku/ruby --buildpack heroku/procfile --buildpack paketo-buildpacks/image-labels -t nickhammond/hotdonuts:eca189d62a8c2ba97fdfca5f85699a50a6d50ce4 -t nickhammond/hotdonuts:latest --env BP_IMAGE_LABELS=service=hotdonuts --path . && docker push nickhammond/hotdonuts:eca189d62a8c2ba97fdfca5f85699a50a6d50ce4 && docker push nickhammond/hotdonuts:latest as n@localhost

[nickhammond]

Highlighting some build time specs with a traditional build with a Dockerfile vs. buildpacks.

I ran these on an M2 Max, 64GB, Docker is capped at 4GB RAM and 4 CPUs. All timing is based on running time bundle exec kamal build push. Using context: "." since I'm just testing this locally, the final image runs in both scenarios.

Buildpacks

builder:
  arch: amd64
  context: "."
  pack:
    builder: "heroku/builder:24"
    buildpacks:
      - heroku/ruby
      - heroku/procfile

Full pack command that Kamal is running

/usr/bin/env pack build nickhammond/hey --platform linux/amd64 \
--creation-time now \
--builder heroku/builder:24 \
--buildpack heroku/ruby \
--buildpack heroku/procfile \
--buildpack paketo-buildpacks/image-labels \
-t nickhammond/hey:e2765756a4ba2cd6dc367302b114ec3757a033f0 \
-t nickhammond/hey:latest-production \
--env BP_IMAGE_LABELS=service=hey \
--path . && \
docker push nickhammond/hey:e2765756a4ba2cd6dc367302b114ec3757a033f0 && \
docker push nickhammond/hey:latest-production

• Fresh build with no build cache: 2.31s user 0.83s system 2% cpu 2:05.44 total
• No code changes, just rebuilding: 2.22s user 0.71s system 7% cpu 39.932 total
• Code changes(app/model change): 2.60s user 0.77s system 8% cpu 40.892 total
• Gemfile(added rest-client): 2.24s user 0.67s system 5% cpu 51.597 total
• Compressed size as stated on Docker Hub 231MB

Dockerfile (The one that ships with Rails - MySQL, Ruby 3.3.5, Rails main, included below)

builder:
  arch: amd64
  context: "."

Dockerfile

# syntax=docker/dockerfile:1
# check=error=true

# This Dockerfile is designed for production, not development. Use with Kamal or build'n'run by hand:
# docker build -t main_app .
# docker run -d -p 80:80 -e RAILS_MASTER_KEY=<value from config/master.key> --name main_app main_app

# For a containerized dev environment, see Dev Containers: https://guides.rubyonrails.org/getting_started_with_devcontainer.html

# Make sure RUBY_VERSION matches the Ruby version in .ruby-version
ARG RUBY_VERSION=3.3.5
FROM docker.io/library/ruby:$RUBY_VERSION-slim AS base

# Rails app lives here
WORKDIR /rails

# Install base packages
RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y curl default-mysql-client libjemalloc2 libvips && \
    rm -rf /var/lib/apt/lists /var/cache/apt/archives

# Set production environment
ENV RAILS_ENV="production" \
    BUNDLE_DEPLOYMENT="1" \
    BUNDLE_PATH="/usr/local/bundle" \
    BUNDLE_WITHOUT="development"

# Throw-away build stage to reduce size of final image
FROM base AS build

# Install packages needed to build gems
RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y build-essential default-libmysqlclient-dev git pkg-config && \
    rm -rf /var/lib/apt/lists /var/cache/apt/archives

# Install application gems
COPY Gemfile Gemfile.lock vendor ./

RUN bundle install && \
    rm -rf ~/.bundle/ "${BUNDLE_PATH}"/ruby/*/cache "${BUNDLE_PATH}"/ruby/*/bundler/gems/*/.git && \
    bundle exec bootsnap precompile --gemfile

# Copy application code
COPY . .

# Precompile bootsnap code for faster boot times
RUN bundle exec bootsnap precompile app/ lib/

# Precompiling assets for production without requiring secret RAILS_MASTER_KEY
RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile




# Final stage for app image
FROM base

# Copy built artifacts: gems, application
COPY --from=build "${BUNDLE_PATH}" "${BUNDLE_PATH}"
COPY --from=build /rails /rails

# Run and own only the runtime files as a non-root user for security
RUN groupadd --system --gid 1000 rails && \
    useradd rails --uid 1000 --gid 1000 --create-home --shell /bin/bash && \
    chown -R rails:rails db log storage tmp
USER 1000:1000

# Entrypoint prepares the database.
ENTRYPOINT ["/rails/bin/docker-entrypoint"]

# Start server via Thruster by default, this can be overwritten at runtime
EXPOSE 80
CMD ["./bin/thrust", "./bin/rails", "server"]

• Fresh build with no build cache: 0.91s user 0.72s system 0% cpu 3:07.55 total
• No code changes, just rebuilding: 0.48s user 0.31s system 14% cpu 5.282 total
• Code changes(app/model change): 0.55s user 0.38s system 2% cpu 33.586 total
• Gemfile(added rest-client): 0.64s user 0.46s system 1% cpu 1:27.40 total
• Compressed size as state on Docker Hub 237MB

A few takeaways:

• The main difference between the initial build with a Dockerfile is that it's doing a bunch of apt installs. The Heroku buildpack already has these built and baked in.
• Heroku buildpacks take a hit for fetching the latest buildpack details which is why the "No code change" run is so much faster(39s vs. 5s) with a Dockerfile. @edmorley Is there a way to pin your buildpack version to a SHA so it doesn't try to fetch on each build? I always see this which takes about 10-15 seconds:

 DEBUG [da5ec3f9]       24: Pulling from heroku/builder
 DEBUG [da5ec3f9]       Digest: sha256:eebd3baebb92ac69437aab9a66f0c169be15aaa24c7e67e7ea3664a8442b54fd
 DEBUG [da5ec3f9]       Status: Image is up to date for heroku/builder:24
 DEBUG [da5ec3f9]       24: Pulling from heroku/heroku
 DEBUG [da5ec3f9]       Digest: sha256:e8884bb60ec847d4b461da844dbe7a802f704c88d6fedb1ad9c4d4294765e443
 DEBUG [da5ec3f9]       Status: Image is up to date for heroku/heroku:24

• For the Gemfile change, the default Dockerfile doesn't include a cache volume for gems which results in re-installing all of the gems for the application. This is of course mostly fixable with a cache mount and modifying your bundle install command.

Caching your bundle install with a Dockerfile

RUN --mount=type=cache,id=bld-gem-cache-3-2,sharing=locked,target=/srv/vendor \
    gem install bundler -v 2.4.22 && \
    bundle config set app_config .bundle && \
    bundle config set path /srv/vendor && \
    bundle config set deployment 'true' && \
    bundle config set without 'development test toolbox' && \
    bundle install --jobs 8 && \
    bundle clean && \
    mkdir -p vendor && \
    bundle config set --local path vendor && \
    cp -ar /srv/vendor . && \
    rm -rf vendor/ruby/*/cache vendor/ruby/*/bundler/gems/*/.git && \
    find vendor/ruby/*/gems/ -name "*.c" -delete && \
    find vendor/ruby/*/gems/ -name "*.o" -delete

Heroku's buildpack automatically utilizing a cache for your gems

 DEBUG [796edaf0]       [builder] - Ruby version `3.3.5` from `Gemfile.lock`
 DEBUG [796edaf0]       [builder]   - Using cached Ruby version
 DEBUG [796edaf0]       [builder] - Bundler version `2.5.3` from `Gemfile.lock`
 DEBUG [796edaf0]       [builder]   - Using cached version
 DEBUG [796edaf0]       [builder] - Bundle install
 DEBUG [796edaf0]       [builder]   - Loading cached gems
 DEBUG [796edaf0]       [builder]   - changes detected in '/workspace/Gemfile.lock' and '/workspace/Gemfile'
 DEBUG [796edaf0]       [builder]   - Running `BUNDLE_BIN="/layers/heroku_ruby/gems/bin" BUNDLE_CLEAN="1" BUNDLE_DEPLOYMENT="1" BUNDLE_GEMFILE="/workspace/Gemfile" BUNDLE_PATH="/layers/heroku_ruby/gems" BUNDLE_WITHOUT="development:test" bundle install`
 DEBUG [796edaf0]       [builder]
 DEBUG [796edaf0]       [builder]       Fetching gem metadata from https://rubygems.org/........
 DEBUG [796edaf0]       [builder]       Fetching domain_name 0.6.20240107
 DEBUG [796edaf0]       [builder]       Fetching http-accept 1.7.0
 DEBUG [796edaf0]       [builder]       Fetching mime-types-data 3.2024.1001
 DEBUG [796edaf0]       [builder]       Fetching netrc 0.11.0
 DEBUG [796edaf0]       [builder]       Installing http-accept 1.7.0
 DEBUG [796edaf0]       [builder]       Installing domain_name 0.6.20240107
 DEBUG [796edaf0]       [builder]       Fetching http-cookie 1.0.7
 DEBUG [796edaf0]       [builder]       Installing mime-types-data 3.2024.1001
 DEBUG [796edaf0]       [builder]       Installing netrc 0.11.0
 DEBUG [796edaf0]       [builder]       Fetching mime-types 3.6.0
 DEBUG [796edaf0]       [builder]       Installing http-cookie 1.0.7
 DEBUG [796edaf0]       [builder]       Installing mime-types 3.6.0
 DEBUG [796edaf0]       [builder]       Fetching rest-client 2.1.0
 DEBUG [796edaf0]       [builder]       Installing rest-client 2.1.0
 DEBUG [796edaf0]       [builder]       Bundle complete! 26 Gemfile dependencies, 108 gems now installed.
 DEBUG [796edaf0]       [builder]       Gems in the groups 'development' and 'test' were not installed.
 DEBUG [796edaf0]       [builder]       Bundled gems are installed into `/layers/heroku_ruby/gems`
 DEBUG [796edaf0]       [builder]
 DEBUG [796edaf0]       [builder]   - Done (7.552s)

[edmorley]

• Heroku buildpacks take a hit for fetching the latest buildpack details which is why the "No code change" run is so much faster(39s vs. 5s) with a Dockerfile. @edmorley Is there a way to pin your buildpack version to a SHA so it doesn't try to fetch on each build? I always see this which takes about 10-15 seconds:

Hi! pack build checking for an updated image should take at most 1-2 seconds to make a couple of requests to the registry, not 15? Are you sure there isn't something else at play? (If the local images are up to date, nothing is actually pulled after the update check.)

If needed, the update check itself can be skipped using pack build ... --pull-policy if-not-present - though you would then need to add a separate way to periodically update the builder+run images (unless the nodes are periodically replaced guaranteeing an image pull at least every few days or similar, to pick up any security updates etc).

[nickhammond]

@edmorley You're right, I found the pull-policy setting and set that to if-not-present. When running with that and --verbose though I'm still seeing a bit of a lag. 10-15s is probably a bit high but somewhere between 5-10s where it stalls and then finally mentions Warning: Builder is trusted but additional modules were added; using the untrusted (5 phases) build flow. Is that an expected amount of time to prepare the pack to build?

[edmorley]

Warning: Builder is trusted but additional modules were added; using the untrusted (5 phases) build flow.

In case it helps, there's some backstory on this warning in:

• https://github.com/buildpacks/pack/issues/2221
• https://github.com/buildpacks/pack/issues/2228

Is that an expected amount of time to prepare the pack to build?

If you wanted to track down where the time was spent, pack build supports a --timestamps option, which when combined with --verbose should make it clearer what is taking so long.

Note there is currently a perf issue with the containerd storage backend at the moment too: https://github.com/buildpacks/pack/issues/2272

[nickhammond]

@edmorley Adding the --timestamps option helped. It's not with every pack but when it does happen it looks like it's when attempting to create the registry cache, here's an example with ~14 seconds. Most of the time it's 5 seconds it looks like though, I think either way it's fine but just trying to keep the pack time low. It looks like I don't have the containerd setting enabled locally too just FYI.

2024/11/26 07:55:05.265098 Creating registry cache for github.com//buildpacks/registry-index
2024/11/26 07:55:19.678694 Pulling image docker.io/paketobuildpacks/image-labels@sha256:1fd7d8f00f15ec404c616f3671aec8019f9a5103eac8e3a8aaf0f2e8d4bb883d with platform linux/amd64

pack CLI run

pack build app --platform linux/amd64 --builder heroku/builder:24 --buildpack heroku/deb-packages --buildpack heroku/ruby --buildpack heroku/procfile --buildpack paketo-buildpacks/image-labels -t app/hey:2bb2b2f7d8b40aa287b65698de0efca208f716d9 -t app/hey:latest-production --env BP_IMAGE_LABELS=service=app --path . --timestamps --verbose
2024/11/26 07:55:02.134940 Using project descriptor located at project.toml
2024/11/26 07:55:02.134959 Builder heroku/builder:24 is trusted
2024/11/26 07:55:02.135017 Pulling image index.docker.io/heroku/builder:24 with platform linux/amd64
2024/11/26 07:55:03.092787 24: 2024/11/26 07:55:03.092834 Pulling from heroku/builder
2024/11/26 07:55:03.092869 2024/11/26 07:55:03.101072 Digest: sha256:2324afe304202e81d452bb203eb4edcc7fed682840d0ec3c82f11fdba96cc199
2024/11/26 07:55:03.101126 Status: Image is up to date for heroku/builder:24
2024/11/26 07:55:03.849599 CheckReadAccess succeeded for the run image docker.io/heroku/heroku:24
2024/11/26 07:55:04.319850 CheckReadAccess succeeded for the run image public.ecr.aws/heroku/heroku:24
2024/11/26 07:55:04.319954 Selected run image docker.io/heroku/heroku:24
2024/11/26 07:55:04.319983 Pulling image docker.io/heroku/heroku:24 with platform linux/amd64
2024/11/26 07:55:05.183422 24: 2024/11/26 07:55:05.183475 Pulling from heroku/heroku
2024/11/26 07:55:05.183492 2024/11/26 07:55:05.189156 Digest: sha256:613aa12fc84c16054be6a9501578e06948d76363e2921e4326447fd0e5a770cf
2024/11/26 07:55:05.189195 Status: Image is up to date for heroku/heroku:24
2024/11/26 07:55:05.202342 Downloading buildpack from registry: paketo-buildpacks/image-labels
2024/11/26 07:55:05.202488 Refreshing registry cache for github.com//buildpacks/registry-index
2024/11/26 07:55:05.202501 Validating registry cache for github.com//buildpacks/registry-index
2024/11/26 07:55:05.265098 Creating registry cache for github.com//buildpacks/registry-index
2024/11/26 07:55:19.678694 Pulling image docker.io/paketobuildpacks/image-labels@sha256:1fd7d8f00f15ec404c616f3671aec8019f9a5103eac8e3a8aaf0f2e8d4bb883d with platform linux/amd64

This branch has been updated with the latest from main as well. I'm currently using this branch to deploy 2 apps that I'm actively working on.

[edmorley]

The Creating registry cache line comes from here: https://github.com/buildpacks/pack/blob/92bc87b297695e4ac6baf559bad2efd55aecec1f/internal/registry/registry_cache.go#L173

[nickhammond]

I've updated this with the latest from main. I've been using this branch to deploy a few projects for a few months now and haven't had any issues. Let me know if I can tweak or update anything.

[nickhammond]

The test matrix run for Ruby 3.4.0-preview2 with the current Gemfile is failing for some reason, it's working fine with the same version and the Rails edge gemfile. It's passing for me locally with that test, I'll try to revisit it this week at some point.

Test failure output

Error:
AppTest#test_stop,_start,_boot,_logs,_images,_containers,_exec,_remove:
RuntimeError: Container not healthy after 30 seconds
    test/integration/integration_test.rb:124:in 'IntegrationTest#wait_for_healthy'
    test/integration/integration_test.rb:8:in 'block in <class:IntegrationTest>'
    vendor/bundle/ruby/3.4.0+0/gems/activesupport-8.0.1/lib/active_support/callbacks.rb:406:in 'BasicObject#instance_exec'
    vendor/bundle/ruby/3.4.0+0/gems/activesupport-8.0.1/lib/active_support/callbacks.rb:406:in 'block in ActiveSupport::Callbacks::CallTemplate::InstanceExec0#make_lambda'
    vendor/bundle/ruby/3.4.0+0/gems/activesupport-8.0.1/lib/active_support/callbacks.rb:178:in 'block in ActiveSupport::Callbacks::Filters::Before#call'
    vendor/bundle/ruby/3.4.0+0/gems/activesupport-8.0.1/lib/active_support/callbacks.rb:667:in 'block (2 levels) in ActiveSupport::Callbacks::CallbackChain#default_terminator'
    vendor/bundle/ruby/3.4.0+0/gems/activesupport-8.0.1/lib/active_support/callbacks.rb:666:in 'Kernel#catch'
    vendor/bundle/ruby/3.4.0+0/gems/activesupport-8.0.1/lib/active_support/callbacks.rb:666:in 'block in ActiveSupport::Callbacks::CallbackChain#default_terminator'
    vendor/bundle/ruby/3.4.0+0/gems/activesupport-8.0.1/lib/active_support/callbacks.rb:179:in 'ActiveSupport::Callbacks::Filters::Before#call'
    vendor/bundle/ruby/3.4.0+0/gems/activesupport-8.0.1/lib/active_support/callbacks.rb:558:in 'block in ActiveSupport::Callbacks::CallbackSequence#invoke_before'
    <internal:array>:42:in 'Array#each'
    vendor/bundle/ruby/3.4.0+0/gems/activesupport-8.0.1/lib/active_support/callbacks.rb:558:in 'ActiveSupport::Callbacks::CallbackSequence#invoke_before'
    vendor/bundle/ruby/3.4.0+0/gems/activesupport-8.0.1/lib/active_support/callbacks.rb:108:in 'ActiveSupport::Callbacks#run_callbacks'
    vendor/bundle/ruby/3.4.0+0/gems/activesupport-8.0.1/lib/active_support/testing/setup_and_teardown.rb:41:in 'ActiveSupport::Testing::SetupAndTeardown#before_setup'

bin/test /home/runner/work/kamal/kamal/test/integration/app_test.rb:4

#<Thread:0x00007fa93d08efb0 /home/runner/work/kamal/kamal/vendor/bundle/ruby/3.4.0+0/gems/sshkit-1.23.2/lib/sshkit/backends/connection_pool.rb:52 run> terminated with exception (report_on_exception is true):
/home/runner/work/kamal/kamal/vendor/bundle/ruby/3.4.0+0/gems/sshkit-1.23.2/lib/sshkit/backends/connection_pool.rb:140:in 'block in SSHKit::Backend::ConnectionPool#run_eviction_loop': unexpected invocation: #<AnyInstance:Object>.sleep(5) (Minitest::Assertion)
	from <internal:kernel>:168:in 'Kernel#loop'
	from /home/runner/work/kamal/kamal/vendor/bundle/ruby/3.4.0+0/gems/sshkit-1.23.2/lib/sshkit/backends/connection_pool.rb:136:in 'SSHKit::Backend::ConnectionPool#run_eviction_loop'
	from /home/runner/work/kamal/kamal/vendor/bundle/ruby/3.4.0+0/gems/sshkit-1.23.2/lib/sshkit/backends/connection_pool.rb:52:in 'block in SSHKit::Backend::ConnectionPool#initialize'

[djmb]

Hey @nickhammond - I'm keen to get this into Kamal - I'll be away next week, but when I'm back I'll do a final review. Thanks for your patience!

[nickhammond]

@djmb No worries, enjoy your time off next week!

Ideally https://github.com/buildpacks/pack/issues/2268 that @edmorley opened gets merged in so we don't have to inject the image-labels pack but that looks like it's still a WIP.

[nickhammond]

The Paketo buildpack CLI also recommends adding labels via the image labels buildpack so we might just want to leave the labelling as-is.

https://paketo.io/docs/howto/configuration/#applying-custom-labels

[nickhammond]

@djmb Are there any tests or verification steps that I can add to make this easier for you to review? I'm using this branch to deploy 5 different apps now, a few weekly and the others at least once a month.

[djmb]

Hey @nickhammond - no nothing needed from me. I'm happy to include this in Kamal 2.7 when that is ready to go out

[nickhammond]

@djmb Sounds good, I'll take a look at the kamal-site docs PR again as well this week.

[nickhammond]

@djmb I updated all of the demo apps and redeployed with the latest using this branch, all went well. The build context was removed "." and there's no longer a need for a project.toml so I removed that from the demo apps.

Let me know if I can provide insight on anything else, excited to get people using this!

[gregjotau]

@nickhammond tried this for a spring boot app:

builder:
  pack:
    # This is the official Paketo builder based on Ubuntu Jammy.
    # It includes buildpacks for Java, Maven, Gradle, and more.
    builder: paketobuildpacks/builder-jammy-base

Got this error:

BUILD SUCCESSFUL in 517ms
5 actionable tasks: 5 up-to-date
Build and push app image...
  INFO [89be1e2e] Running docker --version && docker buildx version as user@localhost
  INFO [89be1e2e] Finished in 0.190 seconds with exit status 0 (successful).
  INFO [e2e8e934] Running docker login -u [REDACTED] -p [REDACTED] as user@localhost
  INFO [e2e8e934] Finished in 2.561 seconds with exit status 0 (successful).
Building with uncommitted changes:
 M config/deploy.yml
  INFO [bce6fcd5] Running /usr/bin/env pack builder inspect paketobuildpacks/builder-jammy-base as user@localhost
  INFO [bce6fcd5] Finished in 2.398 seconds with exit status 0 (successful).
  Finished all in 5.2 seconds
  ERROR (NoMethodError): undefined method `<<' for nil

Using Kamal 2.7.0

If you had insight into how to get this working for a Spring boot app that would be great :)

[nickhammond]

@gregjotau It looks like you've defined a builder but you also need a buildpack. I haven't built a spring app but this looks like the one mentioned in the Spring docs(https://docs.spring.io/spring-boot/reference/packaging/container-images/cloud-native-buildpacks.html)

https://github.com/paketo-buildpacks/spring-boot

builder:
  pack:
    # This is the official Paketo builder based on Ubuntu Jammy.
    # It includes buildpacks for Java, Maven, Gradle, and more.
    builder: paketobuildpacks/builder-jammy-base
    buildpacks:
      - paketo-buildpacks/spring-boot

[gregjotau]

@nickhammond never got it to work, so started a separate dicussion instead: https://github.com/basecamp/kamal/discussions/1637

Thanks for the feature! Would be great if we managed to use it 🙏