kind-on-lima-public
kind-on-lima-public copied to clipboard
source code for the blog post at https://baptistout.net/posts/kubernetes-clusters-on-macos-with-loadbalancer-without-docker-desktop/
= Run multiple KinD clusters on Lima
Demo: https://asciinema.org/a/gZysYfnX0XehRhTnKs8p0BTWS?speed=3&i=2
== Prerequisites
This guide now requires MacOS 13.0+ (Ventura).
Download and install the following tools:
- https://kind.sigs.k8s.io/[kind] ->
brew install kind
- https://direnv.net/[direnv] ->
brew install direnv
- https://baptistout.net/posts/kubernetes-clusters-on-macos-with-loadbalancer-without-docker-desktop/#_setup_part_3_the_docker_cli[docker CLI] (not docker desktop, just the CLI) ->
brew install docker
-
Only for Apple Silicon Mac (M1/M2): install Rosetta ->
softwareupdate --install-rosetta --agree-to-license
=== Additional tools softwareupdate --install-rosetta xcode-select --install
== Getting started
=== Clone this repo First, clone the repo:
git clone [email protected]:bcollard/kind-on-lima-public.git
cd kind-on-lima
Take a look at the environment variables in .envrc
and change the value of LIMA_WORKDIR
to the path of the cloned repo on your machine.
Initialize the env vars for this directory:
direnv allow .
=== Using the scripts
Most of the scripts are available with Makefile targets.
List all the targets with
# Safely run `make` to list all the targets
make
=== MacBook preparation
Run the following make target to create a local data dir (LIMA_DATA_DIR
env var) on your machine. A few docker images will be saved locally as archives. This may take a few minutes.
make prepare-mac-host
ls -l $LIMA_DATA_DIR
The data dir is mainly used by the docker registries to cache images.
=== Lima CLI Two methods here:
-
Method #1: install Lima with brew:
brew install lima
-
Method #2: download the Lima CLI binary with the make target (see below):
make dl-install-lima
; then you are prompted for the version to download (ex. "0.14.2"). This will download the lima command-line (limactl
) and save it in yourLIMA_WORKDIR
. Then the Makefile targets will use this binary. This way, the Lima version is pinned and you don't have to worry about upgrading withbrew
.
NOTE: -> In any case, double check that the env vars named LIMA_BIN
and LIMACTL_BIN
in .envrc
are aligned with the method you chose!
=== Manage the Lima VM
The Lima machine that will be used by the following commands is the one defined in the LIMA_INSTANCE
environment variable (see the .envrc
file)
Example: LIMA_INSTANCE=vz
will use the vz.yaml config file for Lima. And all the following commands will be executed on the VM named vz
.
It's recommanded to use the vz
instance (requires MacOS 13.0+) as it's the fastest one. It's based on macos Virtualization.framework. +
People still using MacOS 12 or lower can use the docker
instance which is using QEMU.
# list the Lima machines:
make list-machines
# create (and start) the VM:
make create
# stop the VM:
make stop
# start the VM:
make start
# delete the VM:
make delete
=== Configure the network E2E The following command will:
- configure the network on your MacBook machine to allow access to the Lima VM
- pre-configures the Kind network bridge on the Lima VM so that it's connected to the local docker registries
- configure the network on the Lima VM to allow access from the MacBook host to the KinD clusters
make config-network-end-to-end
Run this command everytime you restart the Lima VM.
=== Prepare the docker images
make prepare-docker-images
=== Spin up a new KinD cluster
#make kind-create <id> <name>
make kind-create 1 1-istio
make kind-create 2 2-gloo-edge
make kind-create 3 3-gloo-mesh-mgmt
make kind-create 4 4-gloo-mesh-cluster1
make kind-create 5 5-gloo-mesh-cluster2
The kind-create
script takes care of:
- creating a new KinD cluster
- configuring MetalLB so that you can use a LoadBalancer service
- configuring the KinD network bridge to be connected to the local docker registries
=== Test connectivity E2E The following command will help you test the connectivity end-to-end.
It will deploy an nginx instance (pod) on the current cluster, expose the nginx service with a Service type LoadBalancer, and then curl the nginx service from the MacBook host.
make test
make clean-test
=== Manage KinD clusters
# list all the kind clusters:
make kind-list
# delete a specific kind cluster with:
make kind-delete 1-istio
# delete all kind clusters with:
make kind-delete-all
== History and WIP
- 2021: Lima v0.8 + QEMU + MetalLB
- 2022: Lima v0.10 + QEMU + MetalLB + registry mirrors + offline mode bootstrap
- 2023: Lima v0.15 + VZ + Cilium CNI + registry mirrors + offline bootstrap + KinD worker nodes