Added support for custom SSL certificate

[kpumuk]

Following the change in kamal-proxy, this MR introduces a configuration option to load custom SSL certificate and the corresponding private key from disk:

proxy:
  ssl: true
  ssl_certificate_path: /data/cert/foo.example.com/fullchain.pem
  ssl_private_key_path: /data/cert/foo.example.com/privkey.pem

Documentation preview:

[yogeshjain999]

I'm a newbie to kamal (and docker) and was wondering what's the recommended way to get those custom PEM files on server.

Copy using kamal+Dockerfile or do it manually ? Maybe better alternative would be to load the values via ENV variables (given kamal-proxy could support it) ?

[kpumuk]

I'm a newbie to kamal (and docker) and was wondering what's the recommended way to get those custom PEM files on server.

The easiest way would be to use pre-proxy-reboot. For example, if you use 1Password to manage secrets:

1. Put both cert.pem and key.pem under the item you use for secrets
2. Create .kamal/hooks/pre-proxy-reboot:

   #!/bin/sh
   
   set -euo pipefail
   
   KAMAL_PROXY_TLS_CERT=$(op read "op://Private/Kamal Demo/cert.pem")
   KAMAL_PROXY_TLS_PRIVATE_KEY=$(op read "op://Private/Kamal Demo/key.pem")
   
   for ip in ${KAMAL_HOSTS//,/ }; do
     ssh -q -T -o BatchMode=yes ubuntu@"${ip}" bash --noprofile <<-EOF
       mkdir -p .kamal/apps/${KAMAL_SERVICE}/tls
       echo '${KAMAL_PROXY_TLS_CERT}' > .kamal/apps/${KAMAL_SERVICE}/tls/cert.pem
       echo "${KAMAL_PROXY_TLS_PRIVATE_KEY}" > .kamal/apps/${KAMAL_SERVICE}/tls/key.pem
   EOF
   done
   

3. Edit config/deploy.yml to mount TLS certificates to Kamal's image and then enable them:

   proxy:
     ssl: true
     host: app.example.com
     ssl_certificate_path: /home/kamal-proxy/.config/certs/cert.pem
     ssl_private_key_path: /home/kamal-proxy/.config/certs/key.pem
     volumes:
       - "/home/ubuntu/.kamal/apps/demo/certs:/home/kamal-proxy/.config/certs"
   

4. Run kamal proxy reboot to deploy

Copy using kamal+Dockerfile or do it manually ? Maybe better alternative would be to load the values via ENV variables (given kamal-proxy could support it) ?

kamal-proxy does not support environment variables.

TODO: Add volumes support to proxy.

[yogeshjain999]

Cool, setting it up via pre-proxy-reboot sounds good approach. Thanks!!

[kpumuk]

@djmb to follow up on the thread in kamal-proxy, here is the documentation update + support for the recent custom TLS cert changes.

[agu-z]

Is it possible to specify a client certificate too? I need this in order to enable CloudFlare's Authenticated Origin Pulls

[mtmckenna]

With this change, would it be possible to remove the ensure_one_host_for_ssl requirement when providing a cert and key?

Context: I'm looking for a way to have end-to-end in-transit encryption in a regulated environment that requires TLS between the load balancer and server node. I'd also like to be able to use multiple app servers.

By providing my own cert to kamal-proxy, I was thinking I should be able to have the load balancer terminate SSL and then re-encrypt the traffic to kamal-proxy, which would use my supplied cert. wdyt?

Thank you!

[kpumuk]

By providing my own cert to kamal-proxy, I was thinking I should be able to have the load balancer terminate SSL and then re-encrypt the traffic to kamal-proxy, which would use my supplied cert. wdyt?

Yep, that's exactly how it would work with custom certificates, and it removes the limitation of one host behind the load balancer.

[mtmckenna]

Yep, that's exactly how it would work with custom certificates, and it removes the limitation of one host behind the load balancer.

That's great! Would ensure_one_host_for_ssl be updated in a separate PR?

https://github.com/basecamp/kamal/blob/607368121e5bf9fbd8bacd9ae399ada650f3c40c/lib/kamal/configuration/role.rb#L152

Thank you!

[kpumuk]

That's great! Would ensure_one_host_for_ssl be updated in a separate PR?

Great catch, I missed this change in the upstream :-) Added a commit to this MR

[qinmingyuan]

I have used with wildcard domain and multi hosts, Works fine. Please merge this PR.

[christo-ph]

I am looking forward to this feature. Could you kindly explain how I can test this without being merged and released?

[qinmingyuan]

I am looking forward to this feature. Could you kindly explain how I can test this without being merged and released?

I have override the kamal command like this, and add the github repo to Gemfile:

Thread.report_on_exception = false
require_relative '../config/boot'
require 'kamal'
require 'kamal_override'

begin
  Kamal::Cli::Main.start(ARGV)
rescue SSHKit::Runner::ExecuteError => e
  puts "  \e[31mERROR (#{e.cause.class}): #{e.message}\e[0m"
  puts e.cause.backtrace if ENV["VERBOSE"]
  exit 1
rescue => e
  puts "  \e[31mERROR (#{e.class}): #{e.message}\e[0m"
  puts e.backtrace if ENV["VERBOSE"]
  exit 1
end

[Rukamakama]

Looking forward to having this functionality available, we are deploying multiple nodes behind a load balancer and would like to secure the transactions between the load balancer and application servers. Thanks !

[junket]

This seems really close! Thank you @kpumuk ❤️ Anything you need to get this over the finish line?

[frenkel]

I'm trying to move this forward. If I understand the feedback correctly the following changes are needed:

• Modify Kamal::Commands::Proxy#run to:
  • mount an extra volume read-only under /home/kamal-proxy/.config/kamal-proxy-apps
• Modify Kamal::Cli::App#boot to:
  • write certificate and key from secrets into files under /home/kamal-proxy/.config/kamal-proxy-apps/#{service}-#{role}-#{destination}/tls/ using something like docker exec kamal-proxy -ti echo "contents" > /home/kamal-proxy/.config/kamal-proxy-apps/#{service}-#{role}-#{destination}/tls/cert.pem
  • make sure permissions are safe for the keys
• Modify Kamal::Cli::App#remove to:
  • remove the certificate and key from /home/kamal-proxy/.config/kamal-proxy-apps/#{service}-#{role}-#{destination}/tls/

I think writing the secrets into files needs to be done from within the container, so that volume cannot be read-only, right?

Let me know if this is the way you would like to proceed and I'll give it a try.

[jcmaciel]

Hey guys, any updates about it?

[treant-prime]

@kpumuk Hi, I'm having a problem. I'm installing this PR with

gem 'kamal', github: 'basecamp/kamal', ref: "refs/pull/969/head"

and directly form your repo

gem 'kamal', git: 'https://github.com/kpumuk/kamal.git', branch: 'custom-ssl'

yet I am getting ERROR (Kamal::ConfigurationError): proxy: unknown keys: ssl_certificate_path, ssl_private_key_path error during kamal deploy both times,

gemfile.lock:

GIT
  remote: https://github.com/basecamp/kamal.git
  revision: ff32dcb0b9e6741a9d5dfbfbff44fd833f908108
  ref: refs/pull/969/head
  specs:
    kamal (2.2.2)
      activesupport (>= 7.0)
      base64 (~> 0.2)
      bcrypt_pbkdf (~> 1.0)
      concurrent-ruby (~> 1.2)
      dotenv (~> 3.1)
      ed25519 (~> 1.2)
      net-ssh (~> 7.0)
      sshkit (>= 1.23.0, < 2.0)
      thor (~> 1.3)
      zeitwerk (~> 2.5)

Is my installation lacking? ssl_certificate_path seems to be included in code, should I change something? Is this feature working?

[viktorianer]

Did you add those keys to the config? I think you are missing those new keys.

[KhanhVanCong]

Why recommending no SSL when behind Cloudflare? #1039

I have the same issue with you. Do you find any ways to fix it?

[kpumuk]

I am going to work on updating this MR with the recent code and suggestions from @djmb and @kevinmcconnell this week.

[khaled-badenjki]

I need this for a project i'm working on. would appreciate any progress made to it

[letItCurl]

I give you conference ticket if it can motivate you :P

I cant really understand how this is done, but, my usecase is that my VM hostname is not available from the web.

So lets encrypt cant sign this.

[letItCurl]

From my understanding, its merged: https://github.com/basecamp/kamal-proxy/pull/17/files

But we cant toggle the feature from kamal, right?

[mhenrixon]

Do you need any help finishing this up @kpumuk? Would love to see this merged.

[acidtib]

i took a stab at it, loading the custom certificate from secrets #1531

proxy:
  ssl:
    certificate_pem: CERTIFICATE_PEM
    private_key_pem: PRIVATE_KEY_PEM

[frenkel]

As #1531 was merged, I think this PR can be closed @djmb.