Configure Kubernetes Dynamic Volume Provisioning With Heketi & GlusterFS

Posted on 336 views

The Dynamic volume provisioning in Kubernetes allows storage volumes to be created on-demand, without manual Administrator intervention. When developers are doing deployments without dynamic provisioning, cluster administrators have to manually make calls to their cloud or storage provider to create new storage volumes, from where the PersistentVolumes are created.


This guide will discuss how you can achieve Dynamic Volume Provisioning on Kubernetes by using GlusterFS distributed storage solution and Heketi RESTful management interface. It is expected you have deployed Heketi and GlusterFS scale-out network-attached storage file system.

For Ceph, check:

How Dynamic Provisioning is configured in Kubernetes

In Kubernetes, dynamic volume provisioning is based on the API object StorageClass from the API group As a cluster administrator, you’ll define as many StorageClass objects as needed, each specifying a volume plugin ( provisioner) that provisions a volume and the set of parameters to pass to that provisioner when provisioning.

So below are the steps you’ll use to configure Dynamic Volume Provisioning on Kubernetes using Gluster and Heketi API.

Setup GlusterFS and Heketi

It is expected you have a running Gluster and Heketi before you continue with configurations on the Kubernetes end. Refer to our guide below on setting them up.

At the moment we only have guide for CentOS, but we’re working on a deployment guide for Ubuntu/Debian systems.

For containerized setup, check: Setup Kubernetes / OpenShift Dynamic Persistent Volume Provisioning with GlusterFS and Heketi

Once the installation is done, proceed to step 2:

Create StorageClass Object on Kubernetes

We need to create a StorageClass object to enable dynamic provisioning for container platform users. The StorageClass objects define which provisioner should be used and what parameters should be passed to that provisioner when dynamic provisioning is invoked.

  • Check your Heketi Cluster ID
$ heketi-cli cluster list
Id:b182cb76b881a0be2d44bd7f8fb07ea4 [file][block]
  • Create Kubernetes Secret

Get a base64 format of your Heketi admin user password.

$ echo -n "PASSWORD" | base64

Then create a secret with the password for accessing Heketi.

$ vim gluster-secret.yaml
apiVersion: v1
kind: Secret
  name: heketi-secret
  namespace: default
type: ""
  # echo -n "PASSWORD" | base64
  key: cGFzc3dvcmQ=


  • cGFzc3dvcmQ= is the output of echo command.

Create the secret by running the command:

$ kubectl create -f gluster-secret.yaml

Confirm secret creation.

$ kubectl get secret
NAME                                               TYPE                                  DATA   AGE
heketi-secret                                           1      1d
  • Create StorageClass

Below is a sample StorageClass for GlusterFS using Heketi.

$ cat glusterfs-sc.yaml
 kind: StorageClass
   name: gluster-heketi
 reclaimPolicy: Delete
 volumeBindingMode: Immediate
 allowVolumeExpansion: true
   resturl: "http://heketiserverip:8080" 
   restuser: "admin" 
   secretName: "heketi-secret"
   secretNamespace: "default"
   volumetype: "replicate:2"
   volumenameprefix: "k8s-dev"
   clusterid: "b182cb76b881a0be2d44bd7f8fb07ea4"


  • gluster-heketi is the name of the StorageClass to be created.
  • The valid options for reclaim policy are RetainDelete or Recycle. The Delete policy means that a dynamically provisioned volume is automatically deleted when a user deletes the corresponding PersistentVolumeClaim.
  • The volumeBindingMode field controls when volume binding and dynamic provisioning should occur. Valid options are Immediate & WaitForFirstConsumer. The Immediate mode indicates that volume binding and dynamic provisioning occurs once the PersistentVolumeClaim is created. The WaitForFirstConsumer mode delays the binding and provisioning of a PersistentVolume until a Pod using the PersistentVolumeClaim is created.
  • The resturl is the URL of your heketi endpoint
  • heketi-secret is the secret created for Heketi credentials.
  • default is the name of namespace where secret was created
  • replicate:2 indicated the default replication factor for Gluster Volumes created. For more HA, use 3.
  • volumenameprefix: By default dynamically provisioned volumes have the naming schema of vol_UUID format. We have provided a desired volume name from storageclass. So the naming scheme will be: volumenameprefix_Namespace_PVCname_randomUUID
  • b182cb76b881a0be2d44bd7f8fb07ea4 is the ID of the cluster obtained from the command heketi-cli cluster list

Another parameter that can be set is:

volumeoptions: " strict"

The default setting/behavior is:

volumeoptions: " none"

This forces Heketi to strictly place replica bricks in different zones. The required minimum number of nodes required to be present in different zones is if the replica value is set to 3.

Once the file is created, run the following command to create the StorageClass object.

$ kubectl create -f gluster-sc.yaml

Confirm StorageClass creation.

$ kubectl get sc
NAME                            PROVISIONER                    AGE
glusterfs-heketi              1d
local-storage            30d

Step 2: Create PersistentVolumeClaim Object

When a user is requesting dynamically provisioned storage, a storage class should be included in the PersistentVolumeClaim.

Let’s create a 1GB request for storage:

$ vim glusterfs-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
 name: gluster-pvc
 annotations: gluster-heketi
  - ReadWriteMany
     storage: 1Gi

Create object:

$ kubectl create --save-config -f glusterfs-pvc.yaml


$ kubectl get  pvc
NAME             STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
glusterfs-pvc    Bound     pvc-34b9b5e9-fbde-11e9-943f-00505692ee7e   1Gi        RWX            glusterfs-heketi  1d

After creation, you can use it in your deployments. To use the volume we reference the PVC in the YAML file of any Pod/Deployment like this for example:

apiVersion: v1
kind: Pod
  name: gluster-pod
    name: gluster-pod
  - name: gluster-pod
    image: busybox
    command: ["sleep", "60000"]
    - name: gluster-vol
      mountPath: /usr/share/busybox 
      readOnly: false
  - name: gluster-vol
      claimName: glusterfs-pvc

That’s it for today. You should have a working Dynamic Volume Provisioning With Heketi & GlusterFS for your Kubernetes platform.

Gravatar Image
A systems engineer with excellent skills in systems administration, cloud computing, systems deployment, virtualization, containers, and a certified ethical hacker.