Backup Etcd data on OpenShift 4.x to AWS S3 Bucket

Posted on 115 views

You have a running OpenShift Cluster powering your production microservices and worried about etcd data backup?. In this guide we show you how to easily backup etcd and push the backup data to AWS S3 object store. An etcd is a key-value store for OpenShift Container Platform, which persists the state of all resource objects.

In any OpenShift Cluster administration, it is a good and recommended practice to back up your cluster’s etcd data regularly and store it in a secure location. The ideal location for data storage is outside the OpenShift Container Platform environment. This can be an NFS server share, secondary server in your Infrastructure or in a Cloud environment.

The other recommendation is taking etcd backups during non-peak usage hours, as the action is blocking in nature. Ensure etcd backup operation is performed after any OpenShift Cluster upgrade. The importance of this is that during cluster restoration, an etcd backup taken from the same z-stream release must be used. As an example, an OpenShift Container Platform 4.6.3 cluster must use an etcd backup that was taken from 4.6.3.

Step 1: Login to one Master Node in the Cluster

The etcd cluster backup has to be performed on a single invocation of the backup script on a master host. Do not take a backup for each master host.

Login to one master node either through SSH or  debug session:

# SSH Access
$ ssh [email protected]

# Debug session
$ oc debug node/

For a debug session you need to change your root directory to the host:

sh-4.6# chroot /host

If the cluster-wide proxy is enabled, be sure that you have exported the NO_PROXYHTTP_PROXY, and HTTPS_PROXY environment variables.

Step 2: Perform etcd Backup on OpenShift 4.x

An OpenShift cluster access as a user with the cluster-admin role is required to perform this operation.

Before you proceed check to confirm if proxy is enabled:

$ oc get proxy cluster -o yaml

If you have proxy enabled, httpProxyhttpsProxy, and noProxy fields will have the values set.

Run the script to initiate etcd backup process. You should pass a path where backup is saved.

$ mkdir /home/core/etcd_backups
$ sudo /usr/local/bin/ /home/core/etcd_backups

Here is my command execution output:

etcdctl version: 3.3.18
API version: 3.3
found latest kube-apiserver-pod: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-115
found latest kube-controller-manager-pod: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-24
found latest kube-scheduler-pod: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-26
found latest etcd-pod: /etc/kubernetes/static-pod-resources/etcd-pod-11
Snapshot saved at /home/core/etcd_backups/snapshot_2021-03-16_134036.db
snapshot db and kube resources are successfully saved to /home/core/etcd_backups

List files in the backup directory:

$ ls -1 /home/core/etcd_backups/

$ du -sh /home/core/etcd_backups/*
1.5G	/home/core/etcd_backups/snapshot_2021-03-16_134036.db
76K	/home/core/etcd_backups/static_kuberesources_2021-03-16_134036.tar.gz

There will be two files in the backup:

  • snapshot_.db: This file is the etcd snapshot.
  • static_kuberesources_.tar.gz: This file contains the resources for the static pods. If etcd encryption is enabled, it also contains the encryption keys for the etcd snapshot.

Step 3: Push the Backup to AWS S3 (From Bastion Server)

Login from Bastion Server and copy backup files.

scp -r [email protected]:/home/core/etcd_backups ~/

Install AWS CLI tool:

curl "" -o ""

Install unzip tool:

sudo yum -y install unzip

Extract downloaded file:


Install AWS CLI:

$ sudo ./aws/install
You can now run: /usr/local/bin/aws --version

Confirm installation by checking the version:

$ aws --version
aws-cli/2.1.30 Python/3.8.8 Linux/3.10.0-957.el7.x86_64 exe/x86_64.rhel.7 prompt/off

Create OpenShift Backups bucket:

$ aws s3 mb s3://openshiftbackups
make_bucket: openshiftbackups

Create an IAM User:

$ aws iam create-user --user-name backupsonly

Create AWS Policy for Backups user – user able to write to S3 only:

cat >aws-s3-uploads-policy.json<

Apply the policy:

aws iam create-policy --policy-name upload-only-policy --policy-document file://aws-s3-uploads-policy.json

Assign AWS Policy to IAM User:

aws iam attach-user-policy --policy-arn arn:aws:iam:::policy/upload-only-policy --user-name backupsonly

You can now create an access key for an IAM user to test:

$ aws iam create-access-key --user-name backupsonly

        "UserName": "backupsonly",
        "AccessKeyId": "AKIATWFKCYAHF74SCFEP",
        "Status": "Active",
        "SecretAccessKey": "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",
        "CreateDate": "2021-03-16T12:14:39+00:00"

Take note of the Access and Secret Key IDs and use it in configuration:

$ aws configure # On OCP Bastion server


  • AWS Access Key ID
  • AWS Secret Access Key
  • Default region

Try uploading the files to S3 bucket:

$ aws s3 cp etcd_backups/ s3://openshiftbackups/etcd --recursive
upload: etcd_backups/static_kuberesources_2021-03-16_134036.tar.gz to s3://openshiftbackups/etcd/static_kuberesources_2021-03-16_134036.tar.gz
upload: etcd_backups/snapshot_2021-03-16_134036.db to s3://openshiftbackups/etcd/snapshot_2021-03-16_134036.db


$ aws s3 ls s3://openshiftbackups/etcd/
2021-03-16 16:00:59 1549340704 snapshot_2021-03-16_134036.db
2021-03-16 16:00:59      77300 static_kuberesources_2021-03-16_134036.tar.gz

Step 4: Automated Backups to AWS S3 (From Bastion Server)

We can do a script which will perform the following:

  1. Login from bastion to master node
  2. Initiate backup of etcd
  3. Copy backup data from master node to the bastion server
  4. Delete backup data on master node
  5. Copy backup data to the S3 bucket
  6. Delete local data upon successful upload to S3

Create script file on Bastion server:

$ vim

Here is the script which can be modified further for more advanced use case.


# Create backups directory if doesn't exist
[ -d $BACKUPS_DIR ] && echo "Directory Exists" || mkdir $BACKUPS_DIR

# Login and run backup
ssh [email protected]$MASTER_NAME 'mkdir /home/core/etcd_backups' 2>/dev/null
ssh [email protected]$MASTER_NAME 'sudo /usr/local/bin/ /home/core/etcd_backups'
scp -r [email protected]$MASTER_NAME:/home/core/etcd_backups/*  $BACKUPS_DIR/

# clean etcd backups directory on the master node
if [ $RESULT -eq 0 ]; then
    ssh [email protected]$MASTER_NAME 'rm -rf /home/core/etcd_backups/*'

# Backup to aws s3
aws s3 cp $BACKUPS_DIR/ s3://$S3_BUCKET --recursive
# List bucket contents
aws s3 ls s3://$S3_BUCKET/

# Clean backups older than 1 day
#find $BACKUPS_DIR/ -mtime +1 -exec rm  \;
find $BACKUPS_DIR/ -type f -mtime +1 -delete

Using Cron Job that runs 3am:

$ crontab -e
0 3 * * * /path/to/


In this article we’ve looked at how you can backup OpenShift etcd and push the data to an S3 bucket. In our next guide we could discuss on how you can perform a restore from the backup.


Gravatar Image
A systems engineer with excellent skills in systems administration, cloud computing, systems deployment, virtualization, containers, and a certified ethical hacker.