Use Vault-Agent sidecar to inject Secrets in Vault to Kubernetes Pod

Posted on 74 views

In our previous post, we managed to install Vault cluster in GKE and we were not able to add Kubernetes Authentication to it. This post takes you into a different vehicle that will take us through the adding Kubernetes Auth and also add a sample application that we intend to get credentials from Vault. We shall proceed step by step and by the end of the guide, we should have the secrets injected into our Pod in Kubernetes (GKE).

Project Pre-requisites

  • A working Kubernetes Cluster
  • Vault already installed, unsealed and running.

Let us begin

Step 1: Add Kubernetes Authentication Method to Vault.

The token is the one we had after unsealing the Vault leader in the cluster in the previous post.

We shall first exec into the vault-0 container:

$ kubectl exec -it vault-0 -n vault /bin/sh

/ $

Then in the shell prompt that ensues, login to Vault and enable Kubernetes Auth. Use the root token to login to vault.

/ $ vault login

Token (will be hidden): 

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                s.scramblskdfhwkeuigpb
token_accessor       lskdfhwm3zh9blskdfhw0E
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

List raft peers just to confirm everything is okay.

/ $ vault operator raft list-peers

Node                                    Address                        State       Voter
----                                    -------                        -----       -----
71269676-b59e-301a-c054-86a844228bca    vault-0.vault-internal:8201    leader      true
6e0d5183-3896-a73b-be7c-300a61b8a218    vault-1.vault-internal:8201    follower    true
ee238252-5aa5-e5ed-730d-ccb8d6334b1d    vault-2.vault-internal:8201    follower    true
d24854db-05f8-a3ec-f32a-d4e5897ea8db    vault-3.vault-internal:8201    follower    true
c0c774c8-5737-a3c7-cbd7-a838249fafee    vault-4.vault-internal:8201    follower    true

Now enable Kubernetes Auth method

/ $ vault auth enable kubernetes

Success! Enabled kubernetes auth method at: kubernetes/

Step 2: Configure Kubernetes Auth Method

In this step, we are going to bootstrap Kubernetes auth method we enabled in Step 1 by adding relevant configuration details. We will be running commands inside a Vault pod running in Kubernetes.

You will optionally need the following variables:

# JWT is a service account token that has access to the Kubernetes TokenReview API
# You can retrieve this from inside a pod at: /var/run/secrets/
JWT=$(cat /var/run/secrets/

# Address of Kubernetes itself as viewed from inside a running pod

# Kubernetes internal CA
KUBERNETES_CA_CERT=$(cat /var/run/secrets/

Exec/Login into the Vault pod:

kubectl exec -it vault-0 /bin/sh

Then run the following command to configure the Kubernetes Auth Method:

/ $ vault write auth/kubernetes/config \
    token_reviewer_jwt="$(cat /var/run/secrets/" \
    kubernetes_host=https://$KUBERNETES_PORT_443_TCP_ADDR:443 \
    [email protected]/var/run/secrets/ disable_iss_validation=true \

Success! Data written to: auth/kubernetes/config

The “token_reviewer_jwt” and “kubernetes_ca_cert” are mounted to the container by Kubernetes when it is created. The environment variable KUBERNETES_PORT_443_TCP_ADDR is defined and references the internal network address of the Kubernetes host.

Step 3: Add secrets, policy and role to Vault

In this Step, we are going to add a secret to a path in Vault which we shall enable soon. We shall then create a policy that grants specific capabilities to the path such as read, write and others. Some sort of permissions governing how the secret will be accessed. We shall thereafter create a role that will use the policy having the requisite permissions. Let us do this:

First, we shall enable or create a path (secret in this example) that will hold our secrets

/ $ vault secrets enable -path=secret kv-v2

Success! Enabled the kv-v2 secrets engine at: secret/

After that, let us write a sample username and password secret to the path

/ $ vault kv put secret/devwebapp/config username='giraffe' password='salsa'

Key              Value
---              -----
created_time     2021-11-11T14:38:09.892584386Z
deletion_time    n/a
destroyed        false
version          1

Make sure that the username and password was written accordingly by listing

/ $ vault kv get secret/devwebapp/config

====== Metadata ======
Key              Value
---              -----
created_time     2021-11-11T14:38:09.892584386Z
deletion_time    n/a
destroyed        false
version          1

====== Data ======
Key         Value
---         -----
password    salsa
username    giraffe

Beautiful, our username and password was updated successfully.

Next, you remember the policy with permissions we mentioned? We are going to create it now. We will call the policy “demoapp” And as you can see, it references the path we created earlier.

/ $ vault policy write demoapp - <

Success! Uploaded policy: demoapp

Then, the policy will be useless if there are no clients to use it. Roles are the objects that policies are applied to. Another thing about roles is that they are attached to specific Kubernetes service account that ensures that only specific service accounts in Kubernetes cluster will be able to receive the secret stored in Vault. So let us create a role, apply the policy to it and assign it to a service account. Do not worry if the service account is not yet provisioned in your Kubernetes Cluster, we will create it later.

Briefly, the role “devweb-app” connects a Kubernetes service account, “internal-app“, and namespace “vault“, with the Vault policy, “demoapp“.

Here we go:

/ $ vault write auth/kubernetes/role/devweb-app \
        bound_service_account_names=internal-app \
        bound_service_account_namespaces=vault \
        policies=demoapp \
Success! Data written to: auth/kubernetes/role/devweb-app

We have set the time for the token after authentication to expire in 24 hours. You will also notice that we have specified the namespace that the service account will reside in.

You can now exit the vault container terminal

Step 4: Create Service Account and Sample application.

Create a Kubernetes service account named internal-app if it does not exist already in vault namespace.

$ kubectl create sa internal-app -n vault

Create your deployment file with vault’s details added. An example is shown below that will deploy Nginx and add inject the secret in the Vault cluster right into the pod via a sidecar.

As you can see, we have added “” annotations with the details we had set before in Vault such as the name of the role “devweb-app“, the serviceAccount, the secret path and that we want a side-car injected.

$ vim nginx-sample-deployemnt.yaml

apiVersion: apps/v1
kind: Deployment
  name: nginx-vault-demo
  namespace: vault
    app: nginx
      app: nginx
  replicas: 1
      annotations: 'true' 'devweb-app' "false" 'secret/data/devwebapp/config'
        app: nginx
      serviceAccountName: internal-app
        - name: nginx
          image: nginx:latest

Apply the manifest to Kubernetes

$ kubectl apply -f nginx-sample-deployemnt.yaml

Check if the vault-agent pods injected the credentials to your pod

$ kubectl logs nginx-vault-demo-86c599c6cd-8hch9 -n vault -c vault-agent

==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.8.4
             Version Sha: 925bc650ad1d997e84fbb832f302a6bfe0105bbb

2021-11-11T20:20:50.506Z [INFO]  sink.file: creating file sink
2021-11-11T20:20:50.506Z [INFO]  sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2021-11-11T20:20:50.507Z [INFO]  template.server: starting template server
2021-11-11T20:20:50.507Z [INFO] (runner) creating new runner (dry: false, once: false)
2021-11-11T20:20:50.507Z [INFO] (runner) creating watcher
2021-11-11T20:20:50.507Z [INFO]  auth.handler: starting auth handler
2021-11-11T20:20:50.508Z [INFO]  auth.handler: authenticating
2021-11-11T20:20:50.508Z [INFO]  sink.server: starting sink server
2021-11-11T20:20:50.560Z [INFO]  auth.handler: authentication successful, sending token to sinks
2021-11-11T20:20:50.561Z [INFO]  auth.handler: starting renewal process
2021-11-11T20:20:50.561Z [INFO]  sink.file: token written: path=/home/vault/.vault-token
2021-11-11T20:20:50.561Z [INFO]  template.server: template server received new token
2021-11-11T20:20:50.561Z [INFO] (runner) stopping
2021-11-11T20:20:50.561Z [INFO] (runner) creating new runner (dry: false, once: false)
2021-11-11T20:20:50.568Z [INFO] (runner) creating watcher
2021-11-11T20:20:50.568Z [INFO] (runner) starting
2021-11-11T20:20:50.579Z [INFO]  auth.handler: renewed auth token
2021-11-11T20:20:50.684Z [INFO] (runner) rendered "(dynamic)" => "/vault/secrets/config.txt"

As you can see, the runner “rendered” the config file dynamically to the pod

Let us login to the container and check the configuration file under “/vault/secrets/config.txt”

$ kubectl exec -it nginx-vault-demo-86c599c6cd-8hch9 -n vault /bin/sh

# ls /vault/secrets

And there we have our credentials file. We can “cat” it and check if it has the credentials we actually configured in the Vault cluster

# cat /vault/secrets/config.txt
data: map[password:salsa username:giraffe]
metadata: map[created_time:2021-11-11T14:38:09.892584386Z deletion_time: destroyed:false version:1]

And everything is amazingly awesome.

Step 5: Create a template within the secrets file

As you can see from the previous Step, Vault writes the secret in a manner that can be hard for applications to read. Luckily, there is an annotation we can add that will format it in a way that we want it. We will simply add one more annotation as follows:

$ vim nginx-sample-deployemnt.yaml

apiVersion: apps/v1
kind: Deployment
  name: nginx-vault-demo
  namespace: vault
    app: nginx
      app: nginx
  replicas: 1
      annotations: 'true' 'devweb-app' "false" 'secret/data/devwebapp/config' |
          - with secret "secret/data/devwebapp/config" -
          - end -
        app: nginx
      serviceAccountName: internal-app
        - name: nginx
          image: nginx:latest

Now that the edits are done, we can proceed to apply this new manifest to Kubernetes as usual

$ kubectl apply -f nginx-sample-deployemnt.yaml
deployment.apps/nginx-vault-demo configured

View new pod

$ kubectl get pods -n vault 

NAME                                    READY   STATUS        RESTARTS   AGE
nginx-vault-demo-64fb65bb4-g5n9n        2/2     Running       0          8s
vault-0                                 1/1     Running       0          6d19h
vault-1                                 1/1     Running       0          6d19h
vault-2                                 1/1     Running       0          6d19h
vault-3                                 1/1     Running       0          6d19h
vault-4                                 1/1     Running       0          6d19h
vault-agent-injector-7b59447b69-bh9sv   1/1     Running       0          4d15h

You can now exec into the new pod and check if the file was updated accordingly.

$ kubectl exec -it nginx-vault-demo-64fb65bb4-g5n9n -n vault /bin/bash
[email protected]:/# more /vault/secrets/config.txt


Beautiful. Now you can extract those fields that are formatted properly and use them in your application

End Note

We have managed to use the Vault Cluster we built earlier by using a real application that received secrets saved far away in Vault. Now you can enjoy the separation of concerns and know that your secrets are well kept in Vault. There is no need anymore to upload your env files to git. Just place all of the secrets in Vault and you can use the Sidecar model or you can code the logic directly so that your application can retrieve the secrets from Vault directly.

We thank your readership, your encouraging comments and the enormous support pouring in. Have a joyous end of the year season guys!!


Gravatar Image
A systems engineer with excellent skills in systems administration, cloud computing, systems deployment, virtualization, containers, and a certified ethical hacker.