You want to deploy a Lightweight Kubernetes Cluster with ease and less memory footprint?, all in a binary less than 40mb?. Kubernetes has been a game changer in how containerized workloads are deployed and managed at immense scale. The main challenge for Developers revolve around setup process and resource requirements to have a working Kubernetes Cluster.
For Production deployment refer to:
- Deploy Production Ready Kubernetes Cluster with Ansible & Kubespray
- Install Production Kubernetes Cluster with Rancher RKE
For semi manual deployment with Kubeadm, check:
For Development and test impetus, a user should be able to deploy Kubernetes with least resource utilization and low hardware specs. K3s is a tiny Kubernetes distribution created and optimized by Rancher team to run on IoT and ARM powered devices.
Since K3s is optimized to use less resources, some Kubernetes features are stripped off. These include:
- Legacy, alpha, non-default features
- Most in-tree plugins (cloud providers and storage plugins) which can be replaced with out of tree addons.
- etcd3 replaced by sqlite3 as the default storage mechanism
Deploy Lightweight Kubernetes Cluster in 5 minutes with K3s
For my deployment, I have three servers running Debian 10 each with 1GB of Ram and 1vcpu. One of the servers will be used as master and other two as worker nodes.
$ openstack server list
+--------------------------------------+-------------------+---------+-----------------------------------+-----------+-----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-------------------+---------+-----------------------------------+-----------+-----------+
| 4df6a6dc-26e8-4ae0-8b6e-2f97daec0ef3 | k3s-master | ACTIVE | private=10.10.1.159 | Debian-10 | m1.tiny |
| 5ca13239-b745-4f62-ab11-0a27949c9b35 | k3s-node02 | ACTIVE | private=10.10.1.142 | Debian-10 | m1.tiny |
| a54997f2-4d94-4718-86ab-73609b328761 | k3s-node01 | ACTIVE | private=10.10.1.126 | Debian-10 | m1.tiny |
+--------------------------------------+-------------------+---------+-----------------------------------+-----------+-----------+I’ll add A records for the servers on /etc/hosts file on each server.
$ sudo vim /etc/hosts
10.10.1.159 k3s-master
10.10.1.126 k3s-node01
10.10.1.142 k3s-node02Install K3s on Master node
There are many ways to run k3s. The quickest method is installation via provided bash script. This script provides a convenient way for installing to systemd or openrc.
curl -sfL https://get.k3s.io | sh -Installation output.
[INFO] Finding release for channel stable
[INFO] Using v1.24.4+k3s1 as release
[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.24.4+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.24.4+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Skipping installation of SELinux RPM
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3sThe service is automatically started after installation.
$ systemctl status k3s
● k3s.service - Lightweight Kubernetes
Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-09-06 16:08:08 EAT; 14s ago
Docs: https://k3s.io
Process: 363618 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
Process: 363620 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
Process: 363621 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Main PID: 363622 (k3s-server)
Tasks: 29
Memory: 567.1M
CPU: 10.727s
CGroup: /system.slice/k3s.service
├─363622 "/usr/local/bin/k3s server"
└─363642 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/contai>
Sep 06 16:08:22 jammy k3s[363622]: W0906 16:08:22.168575 363622 controllermanager.go:558] "cloud-node-lifecycle" is disabled
Sep 06 16:08:22 jammy k3s[363622]: I0906 16:08:22.168816 363622 deployment_controller.go:153] "Starting controller" controller="deployment"
Sep 06 16:08:22 jammy k3s[363622]: I0906 16:08:22.169005 363622 shared_informer.go:255] Waiting for caches to sync for deployment
Sep 06 16:08:22 jammy k3s[363622]: I0906 16:08:22.318925 363622 controllermanager.go:593] Started "persistentvolume-expander"
Sep 06 16:08:22 jammy k3s[363622]: I0906 16:08:22.319321 363622 expand_controller.go:341] Starting expand controller
Sep 06 16:08:22 jammy k3s[363622]: I0906 16:08:22.319473 363622 shared_informer.go:255] Waiting for caches to sync for expand
Sep 06 16:08:22 jammy k3s[363622]: I0906 16:08:22.625454 363622 controllermanager.go:593] Started "horizontalpodautoscaling"
Sep 06 16:08:22 jammy k3s[363622]: W0906 16:08:22.626280 363622 controllermanager.go:558] "route" is disabled
Sep 06 16:08:22 jammy k3s[363622]: I0906 16:08:22.626475 363622 horizontal.go:168] Starting HPA controller
Sep 06 16:08:22 jammy k3s[363622]: I0906 16:08:22.626632 363622 shared_informer.go:255] Waiting for caches to sync for HPAA kubeconfig file is written to /etc/rancher/k3s/k3s.yaml:
$ sudo cat /etc/rancher/k3s/k3s.yaml
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkekNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUyTmpJME5qazJPREl3SGhjTk1qSXdPVEEyTVRNd09EQXlXaGNOTXpJd09UQXpNVE13T0RBeQpXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUyTmpJME5qazJPREl3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFUQldzaU5RNHpUZU1Dc3pxd2hzWTRDdExnc0V3WE9MajJ4VEtHRlNjYS8KWVdYM3hyNFp0TTFJcENXSXAwNjlMcDRtWUo5MUlyWm8zSDdCQmtaQzU5TXNvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVWNLekhBYkd3cXJoZVFOQzZpdG56CnVYNGJ3bHd3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUloQU9qOVBlTkNOaXdvcHhPc2N6Rkgzd1hUT2dHdDl5eUcKR0RKNmR5bXljM29HQWlBTjJXUUtSQ2MzNkVyY0NacS8rMHJZcGEraEJKZVFVbGtBcFZ1bThFcVFLQT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
server: https://127.0.0.1:6443
name: default
contexts:
- context:
cluster: default
user: default
name: default
current-context: default
kind: Config
preferences:
users:
- name: default
user:
client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrRENDQVRlZ0F3SUJBZ0lJVXVpb0JCVDRSVGt3Q2dZSUtvWkl6ajBFQXdJd0l6RWhNQjhHQTFVRUF3d1kKYXpOekxXTnNhV1Z1ZEMxallVQXhOall5TkRZNU5qZ3lNQjRYRFRJeU1Ea3dOakV6TURnd01sb1hEVEl6TURrdwpOakV6TURnd01sb3dNREVYTUJVR0ExVUVDaE1PYzNsemRHVnRPbTFoYzNSbGNuTXhGVEFUQmdOVkJBTVRESE41CmMzUmxiVHBoWkcxcGJqQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJMT0NUdSs1eHBHRjlqMmcKeWVGK3REYnZKejg2M29wbmtFVUVQL0VOTDV4SEtINVpkQlNpM0pHa1AyTzVNUWNxcU9lOEdMd01lTVpHdG9jNAp2WU9ybWV5alNEQkdNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WSFNVRUREQUtCZ2dyQmdFRkJRY0RBakFmCkJnTlZIU01FR0RBV2dCUVJyanUvVnlvRHg4ek9tWmdVL094UWYwUi8yekFLQmdncWhrak9QUVFEQWdOSEFEQkUKQWlBenlDR29vTERvem5hNVZ0ODB6UnMyT2hkT05KbE5VVnRwWWh4N0RNN0RSQUlnUUozVHZhRitZaGxBY2FjTQpYV1VPNmpmMUZnZkw2MU1qb0VUeGFVbzk5Szg9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkekNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdFkyeHAKWlc1MExXTmhRREUyTmpJME5qazJPREl3SGhjTk1qSXdPVEEyTVRNd09EQXlXaGNOTXpJd09UQXpNVE13T0RBeQpXakFqTVNFd0h3WURWUVFEREJock0zTXRZMnhwWlc1MExXTmhRREUyTmpJME5qazJPREl3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFRc2VQOERKZC96dVpPb0lHazlOUllJR2lmNURIRWhESVp1ZGkvc0RobmYKTkFRNEZ6cUl4U0RaZlFHdEw1UTErU0QzOXBXb1puYTh2aHJPYzZrcVAvbU1vMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVUVhNDd2MWNxQThmTXpwbVlGUHpzClVIOUVmOXN3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUloQVBmQkZxcUN5SlI1YnRIVW4xS3JMKzZOYkwyYXJ3VmwKUGt4alNKVlpOUUE5QWlCQ2RoVHVuL2d1NWw0WHVSSE9Ba2xWb0E3K1ZNVGwvdG05OUZXSEk1M3J4Zz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUp2MEE5UjUramtDRi82RW9NWXZadU9MYmh3eXlXKzZwcEhQWFpWVXdvNUlvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFczRKTzc3bkdrWVgyUGFESjRYNjBOdThuUHpyZWltZVFSUVEvOFEwdm5FY29mbGwwRktMYwprYVEvWTdreEJ5cW81N3dZdkF4NHhrYTJoemk5ZzZ1WjdBPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo=Other utilities installed are:
- kubectl
- crictl
- k3s-killall.sh
- k3s-uninstall.sh
Install K3s on Worker Nodes
To install on worker nodes we should pass K3S_URL along with K3S_TOKEN or K3S_CLUSTER_SECRET environment variables.
The K3S_TOKEN is created at /var/lib/rancher/k3s/server/node-token on first node.
$ sudo cat /var/lib/rancher/k3s/server/node-token
K1037242285167f68f26c05e9221751153591384faab8bb6318c3be56a1ff6286e9::server:8d2fb5bc5ca6eb587de0e5920203309bSo to install Kubernetes on worker nodes I’ll run:
k3s_url="https://k3s-master:6443"
k3s_token="K1037242285167f68f26c05e9221751153591384faab8bb6318c3be56a1ff6286e9::server:8d2fb5bc5ca6eb587de0e5920203309b"
curl -sfL https://get.k3s.io | K3S_URL=$k3s_url K3S_TOKEN=$k3s_token sh -Installation output:
[INFO] Finding latest release
[INFO] Using v1.24.4+k3s1 as release
[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.24.4+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.24.4+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s-agent.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s-agent.service
[INFO] systemd: Enabling k3s-agent unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service.
[INFO] systemd: Starting k3s-agentLogin to one of the master node and check cluster status.
$ sudo kubectl config get-clusters
NAME
default
$ sudo kubectl cluster-info
Kubernetes master is running at https://localhost:6443
CoreDNS is running at https://localhost:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
$ sudo kubectl get nodes
NAME STATUS ROLES AGE VERSION
k3s-master Ready master 14m v1.24.4+k3s1
k3s-node01 Ready worker 3m11s v1.24.4+k3s1
k3s-node02 Ready worker 3m58s v1.24.4+k3s1
$ sudo kubectl get namespaces
NAME STATUS AGE
default Active 16m
kube-node-lease Active 16m
kube-public Active 16m
kube-system Active 16m
$ sudo kubectl get endpoints -n kube-system
NAME ENDPOINTS AGE
kube-dns 10.42.0.2:53,10.42.0.2:53,10.42.0.2:9153 14m
traefik 10.42.0.5:80,10.42.0.5:443 14m
$ sudo kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-b7464766c-q9frk 1/1 Running 0 15m
helm-install-traefik-8dhpk 0/1 Completed 0 15m
svclb-traefik-9c2j8 2/2 Running 0 4m49s
svclb-traefik-bf9zd 2/2 Running 0 4m2s
svclb-traefik-v2fpx 2/2 Running 0 14m
traefik-5c79b789c5-k589d 1/1 Running 0 14mUse crictl command to see running containers.
# Master
$ sudo crictl ps
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT POD ID
acfafb50852d3 18471c10e6e4b 16 minutes ago Running traefik 0 bf8534452389f
fee5ac7e88f2e 4a065d8dfa588 16 minutes ago Running lb-port-443 0 e7068ff7ab2f2
bbab5b07e5efb 4a065d8dfa588 16 minutes ago Running lb-port-80 0 e7068ff7ab2f2
65c5d1333ea04 2ee68ed074c6e 16 minutes ago Running coredns 0 435c51f4716fc
# Workers
$ sudo crictl ps
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT POD ID
7ad5c83d6466f 4a065d8dfa588 6 minutes ago Running lb-port-443 0 bf8d9fe57c3f3
c1380eabc0b33 4a065d8dfa588 6 minutes ago Running lb-port-80 0 bf8d9fe57c3f3Uninstall K3s
The k3s installer script will install k3s and additional utilities, such as kubectl, crictl, k3s-killall.sh, and k3s-uninstall.sh. To uninstall K3s, run the command:
$ k3s-uninstall.shCheck K3s documentation for advanced configurations.
