After a successful installation and configuration of OpenShift Container Platform, the updates are providedover-the-air by OpenShift Update Service (OSUS). The operator responsible for checking valid updates available for your cluster with the OpenShift Update Service is called Cluster Version Operator (CVO). When you request an update, the CVO uses the release image for that update to upgrade your cluster. All the release artifacts are stored as container images in the Quay registry.
It is important to note that the OpenShift Update Service displays all valid updates for your Cluster version. It is highly recommended that you do not force an update to a version that the OpenShift Update Service does not display. This is because a suitability check is performed to guarantee functional cluster after the upgrade. During the upgrade process, the Machine Config Operator (MCO) applies the new configuration to your cluster machines.
Before you start a minot upgrade to your OpenShift Cluster, check the current cluster version using oc command line tool if configured or from a web console. You should have the cluster admin rolebinding to use these functions.
We have the following OpenShift / OKD installation guides on our website:
- How To Deploy OpenShift Container Platform 4.x on KVM
- How To Install OKD OpenShift 4.x Cluster on OpenStack
- Setup Local OpenShift 4.x Cluster with CodeReady Containers
1) Confirm current OpenShift Cluster version
Check the current version and ensure your cluster is available:
$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.5 True False 24d Cluster version is 4.8.5
The current version of OpenShift Container Platform installed can also be checked from the web console – Administration → Cluster Settings > Details
Also check available Cluster nodes and their current status. Ensure they are all in Ready State before you can initiate an upgrade.
$ oc get nodes NAME STATUS ROLES AGE VERSION master01.ocp4.computingpost.com Ready master 24d v1.21.1+9807387 master02.ocp4.computingpost.com Ready master 24d v1.21.1+9807387 master03.ocp4.computingpost.com Ready master 24d v1.21.1+9807387 worker01.ocp4.computingpost.com Ready worker 24d v1.21.1+9807387 worker02.ocp4.computingpost.com Ready worker 24d v1.21.1+9807387 worker03.ocp4.computingpost.com Ready worker 24d v1.21.1+9807387
2) Backup Etcd database data
Access one of the control plane nodes(master node) using oc debug command to start a debug session:
$ oc debug node/
Here is an example with expected output:
$ oc debug node/master01.ocp4.example.com Starting pod/master01ocp4examplecom-debug ... To use host binaries, run `chroot /host` Pod IP: 192.168.100.11 If you don't see a command prompt, try pressing enter. sh-4.4#
Change your root directory to the host:
sh-4.4# chroot /host
Then initiate backup of etcd data using provided script namedcluster-backup.sh:
sh-4.4# which cluster-backup.sh /usr/local/bin/cluster-backup.sh
The cluster-backup.sh script is part of etcd Cluster Operator and it is just a wrapper around the
etcdctl snapshot save command. Execute the script while passing the backups directory:
sh-4.4# /usr/local/bin/cluster-backup.sh /home/core/assets/backup
Here is the output as captured from my backup process
found latest kube-apiserver: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-19 found latest kube-controller-manager: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-8 found latest kube-scheduler: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-9 found latest etcd: /etc/kubernetes/static-pod-resources/etcd-pod-3 3f8cc62fb9dd794113201bfabd8af4be0fdaa523987051cdb358438ad4e8aca6 etcdctl version: 3.4.14 API version: 3.4 "level":"info","ts":1631392412.4503953,"caller":"snapshot/v3_snapshot.go:119","msg":"created temporary db file","path":"/home/core/assets/backup/snapshot_2021-09-11_203329.db.part" "level":"info","ts":"2021-09-11T20:33:32.461Z","caller":"clientv3/maintenance.go:200","msg":"opened snapshot stream; downloading" "level":"info","ts":1631392412.4615548,"caller":"snapshot/v3_snapshot.go:127","msg":"fetching snapshot","endpoint":"https://184.108.40.206:2379" "level":"info","ts":"2021-09-11T20:33:33.712Z","caller":"clientv3/maintenance.go:208","msg":"completed snapshot read; closing" "level":"info","ts":1631392413.9274824,"caller":"snapshot/v3_snapshot.go:142","msg":"fetched snapshot","endpoint":"https://220.127.116.11:2379","size":"102 MB","took":1.477013816 "level":"info","ts":1631392413.9344463,"caller":"snapshot/v3_snapshot.go:152","msg":"saved","path":"/home/core/assets/backup/snapshot_2021-09-11_203329.db" Snapshot saved at /home/core/assets/backup/snapshot_2021-09-11_203329.db "hash":3708394880,"revision":12317584,"totalKey":7946,"totalSize":102191104 snapshot db and kube resources are successfully saved to /home/core/assets/backup
Check if the backup files are available in our backups directory:
sh-4.4# ls -lh /home/core/assets/backup/ total 98M -rw-------. 1 root root 98M Sep 11 20:33 snapshot_2021-09-11_203329.db -rw-------. 1 root root 92K Sep 11 20:33 static_kuberesources_2021-09-11_203329.tar.gz
The files as seen are:
snapshot_: The etcd snapshot file.
static_kuberesources_: File that contains the resources for the static pods. When etcd encryption is enabled, the encryption keys for the etcd snapshot will be contained in this file.
You can copy the backup files to a separate system or location outside the server for better security if the node becomes unavailable during upgrade.
3) Changing Updates Channel (Optional)
The OpenShift Container Platform offers the following upgrade channels:
Review the current update channel information and confirm that your channel is set to
$ oc get clusterversion -o json|jq ".items.spec" "channel": "fast-4.8", "clusterID": "f3dc42b3-aeec-4f4c-980f-8a04d6951585"
You can decide to change an upgrade channel before the actual upgrade of the cluster.
From Command Line Interface
Switch Update channel from CLI using patch:
oc patch clusterversion version --type json -p '["op": "add", "path": "/spec/channel", "value": "
”]' # Example $ oc patch clusterversion version --type json -p '["op": "add", "path": "/spec/channel", "value": "stable-4.8"]' clusterversion.config.openshift.io/version patched $ oc get clusterversion -o json|jq ".items.spec" "channel": "stable-4.8", "clusterID": "f3dc42b3-aeec-4f4c-980f-8a04d6951585"
From Web Console
NOTE:For production clusters, you must subscribe to a stable-* or fast-* channel. Your cluster is fully supported by Red Hat subscription if you change from stable to fast channel.
In my example below I’ve set the channel to fast-4.8.
4) Perform Minor Upgrade on OpenShift / OKD Cluster
You can choose to perform a cluster upgrade from:
- Bastion Server / Workstation oc command line
- From OpenShift web console
Upgrade your OpenShift Container Platform from CLI
Check available upgrades
$ oc adm upgrade Cluster version is 4.8.5 Updates: VERSION IMAGE 4.8.9 quay.io/openshift-release-dev/[email protected]:5fb4b4225498912357294785b96cde6b185eaed20bbf7a4d008c462134a4edfd 4.8.10 quay.io/openshift-release-dev/[email protected]:53576e4df71a5f00f77718f25aec6ac7946eaaab998d99d3e3f03fcb403364db
As seen we have two minor upgrades that can be performed:
- To version 4.8.9
- To version 4.8.10
The easiest way to upgrade is to the latest version:
$ oc adm upgrade --to-latest=true Updating to latest version 4.8.10
To update to a specific version:
$ oc adm upgrade --to=
#e.g 4.8.9, I'll run: $ oc adm upgrade --to=4.8.9
You can easily review Cluster Version Operator status with the following command:
$ oc get clusterversion -o json|jq ".items.spec" "channel": "stable-4.8", "clusterID": "f3dc42b3-aeec-4f4c-980f-8a04d6951585", "desiredUpdate": "force": false, "image": "quay.io/openshift-release-dev/[email protected]:53576e4df71a5f00f77718f25aec6ac7946eaaab998d99d3e3f03fcb403364db", "version": "4.8.10"
oc adm upgrade command will give progress update with the steps:
$ oc adm upgrade info: An upgrade is in progress. Working towards 4.8.10: 69 of 678 done (10% complete) Updates: VERSION IMAGE 4.8.9 quay.io/openshift-release-dev/[email protected]:5fb4b4225498912357294785b96cde6b185eaed20bbf7a4d008c462134a4edfd 4.8.10 quay.io/openshift-release-dev/[email protected]:53576e4df71a5f00f77718f25aec6ac7946eaaab998d99d3e3f03fcb403364db
Upgrade OpenShift Container Platform from UI
Administration → Cluster Settings→ Details→ Select channel→ Select a version to update to, and click Save. The Input channel Update status changes to Update to
All cluster operators will be upgraded one after the other until all are in the minor version selected during upgrade:
$ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.8.5 True False False 119m baremetal 4.8.5 True False False 24d cloud-credential 4.8.5 True False False 24d cluster-autoscaler 4.8.5 True False False 24d config-operator 4.8.5 True False False 24d console 4.8.5 True False False 36h csi-snapshot-controller 4.8.5 True False False 24d dns 4.8.5 True False False 24d etcd 4.8.10 True False False 24d image-registry 4.8.5 True False False 24d ingress 4.8.5 True False False 24d insights 4.8.5 True False False 24d kube-apiserver 4.8.5 True False False 24d kube-controller-manager 4.8.5 True False False 24d kube-scheduler 4.8.5 True False False 24d kube-storage-version-migrator 4.8.5 True False False 4d16h machine-api 4.8.5 True False False 24d machine-approver 4.8.5 True False False 24d machine-config 4.8.5 True False False 24d marketplace 4.8.5 True False False 24d monitoring 4.8.5 True False False
network 4.8.5 True False False 24d node-tuning 4.8.5 True False False 24d openshift-apiserver 4.8.5 True False False 32h openshift-controller-manager 4.8.5 True False False 23d openshift-samples 4.8.5 True False False 24d operator-lifecycle-manager 4.8.5 True False False 24d operator-lifecycle-manager-catalog 4.8.5 True False False 24d operator-lifecycle-manager-packageserver 4.8.5 True False False 7d11h service-ca 4.8.5 True False False 24d storage 4.8.5 True False False 24d
5) Validate OpenShift CLuster Upgrade
Wait for the upgrade process to complete then confirm that the cluster version has updated to the new version:
$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.10 True False 37h Cluster version is 4.8.10
Checking cluster version from the web console
To obtain more detailed information about the cluster status run the command:
$ oc describe clusterversion
If you try running the command
oc adm upgrade immediately after upgrade to the latest release you should get a message similar to below:
$ oc adm upgrade Cluster version is 4.8.10 No updates available. You may force an upgrade to a specific release image, but doing so may not be supported and result in downtime or data loss.
In this short guide we’ve shown how one can easily perform minor upgrade of OpenShift container cluster version. The process can be initiated from a web console or from the command line, it all depends on your preference. In our articles to follow we’ll cover steps required to perform Major versions upgrade in anOpenShift container cluster.