Deploying Prometheus on EKS Kubernetes Cluster

Posted on 128 views

Prometheus is an open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach. Prometheus can be installed as standalone service in a Linux machine or deployed in a Kubernetes cluster. In this guide we will walk you through the installation of Prometheus on an EKS Cluster deployed in AWS Cloud.

The main features of Prometheus are:

  • a multi-dimensional data model with time series data identified by metric name and key/value pairs
  • PromQL, a flexible query language to leverage this dimensionality
  • no reliance on distributed storage; single server nodes are autonomous
  • time series collection happens via a pull model over HTTP
  • pushing time series is supported via an intermediary gateway
  • targets are discovered via service discovery or static configuration
  • multiple modes of graphing and dashboarding support

Deploying Prometheus on EKS Kubernetes Cluster

You should have a running EKS Cluster before you can use this guide with kubectl installed and able to interact with the API server.

List available clusters.

$ eksctl get cluster
prod-eks-cluster	eu-west-1

Step 1: Configure kubectl

To obtain cluster credentials at any point in time from an EKS cluster deployed with eksctl, run:

$ eksctl utils write-kubeconfig --cluster= [--kubeconfig=][--set-kubeconfig-context=]
See below example:
# Create default kubeconfig directory - Can be custom
--- Linux ---
$ mkdir /home/$USER/.kube

--- macOS ---
$ mkdir /Users/$USER/.kube

# Obtain cluster credentials
--- Linux ---
$ s write-kubeconfig --cluster=prod-eks-cluster --kubeconfig=/home/$USER/.kube/config

--- macOS ---
$ eksctl utils write-kubeconfig --cluster=prod-eks-cluster --kubeconfig=/Users/$USER/.kube/config

Expected command execution output:

[ℹ]  eksctl version 0.25.0
[ℹ]  using region eu-west-1
[✔]  saved kubeconfig as "/Users/jkmutai/.kube/config"

Confirm if kubectl command is working as expected.

$ kubectl get  nodes
NAME                                            STATUS   ROLES    AGE     VERSION   Ready       7d21h   v1.17.9-eks-4c6976   Ready       7d21h   v1.17.9-eks-4c6976

Step 2: Install Kubernetes Metrics Server

The Kubernetes API server exposes a number of metrics that are useful for monitoring and analysis. These metrics are exposed internally through a metrics endpoint that refers to the /metrics HTTP API.

Install Metrics Server with the following guide:

Install Kubernetes Metrics Server on Amazon EKS Cluster

You can view cluster raw metrics by running the following command:

$ kubectl get --raw /metrics

Step 3: Install Helm

The Helm package manager for Kubernetes helps you install and manage applications on your Kubernetes cluster. We’ll be using help to deploy Prometheus on EKS.

These are the commands you’ll use to install helm.

--- Linux ---
curl >
chmod 700

--- macOS ---
$ brew install helm

--- Windows ---
$ choco install kubernetes-helm

You can query installed version using the command:

$ helm version
version.BuildInfoVersion:"v3.2.4", GitCommit:"0ad800ef43d3b826f31a5ad8dfbb4fe05d143688", GitTreeState:"dirty", GoVersion:"go1.14.3"

Step 4: Deploying Prometheus on EKS Kubernetes Cluster

Prometheus can be installed on Kubernetes cluster using Operator or with helm. We’ll use Helm V3 we just installed to deploy prometheus in an EKS cluster.

First create a monitoring namespace.

$ kubectl create namespace monitoring
namespace/monitoring created

Prometheus needs a way to persist metrics data for historical reference. We’ll use EBS which is provisioned with gp2 storage class.

$ kubectl get sc
kubectl get sc
efs-sc         Delete          Immediate              false                  2d23h
gp2 (default)   Delete          WaitForFirstConsumer   false                  7d21h

Add chart repository:

$ helm repo add stable
"stable" has been added to your repositories

Deploy Prometheus using Helm.

helm install prometheus stable/prometheus \
    --namespace monitoring \
    --set alertmanager.persistentVolume.storageClass="gp2",server.persistentVolume.storageClass="gp2"

Command execution output:

NAME: prometheus
LAST DEPLOYED: Wed Aug 19 22:01:16 2020
NAMESPACE: monitoring
STATUS: deployed
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:

Confirm PV and PVC are created.

$ kubectl get pv -n monitoring
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                STORAGECLASS   REASON   AGE
pvc-7a1d891c-5b3f-4d28-91b5-5f1d1590daf9   8Gi        RWO            Delete           Bound    monitoring/prometheus-server         gp2                     11m
pvc-7d45951f-a8c5-4fb1-b149-747dccaceecc   2Gi        RWO            Delete           Bound    monitoring/prometheus-alertmanager   gp2                     11m

$ kubectl get pvc -n monitoring
NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
prometheus-alertmanager   Bound    pvc-7d45951f-a8c5-4fb1-b149-747dccaceecc   2Gi        RWO            gp2            11m
prometheus-server         Bound    pvc-7a1d891c-5b3f-4d28-91b5-5f1d1590daf9   8Gi        RWO            gp2            11m

Step 4: Access Prometheus on EKS Kubernetes Cluster

After installation query all resources in the monitoring namespace:

$ kubectl get all -n monitoring
NAME                                                 READY   STATUS    RESTARTS   AGE
pod/prometheus-alertmanager-d47577c4b-kz9td          2/2     Running   0          98s
pod/prometheus-kube-state-metrics-6df5d44568-fzfwn   1/1     Running   0          98s
pod/prometheus-node-exporter-p6qk7                   1/1     Running   0          99s
pod/prometheus-node-exporter-stsjk                   1/1     Running   0          99s
pod/prometheus-pushgateway-57c97d878d-hmnwr          1/1     Running   0          98s
pod/prometheus-server-559c49b4ff-42k46               2/2     Running   0          98s

NAME                                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/prometheus-alertmanager         ClusterIP           80/TCP     99s
service/prometheus-kube-state-metrics   ClusterIP            8080/TCP   99s
service/prometheus-node-exporter        ClusterIP   None                     9100/TCP   99s
service/prometheus-pushgateway          ClusterIP            9091/TCP   99s
service/prometheus-server               ClusterIP            80/TCP     99s

NAME                                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/prometheus-node-exporter   2         2         2       2            2                     99s

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/prometheus-alertmanager         1/1     1            1           99s
deployment.apps/prometheus-kube-state-metrics   1/1     1            1           99s
deployment.apps/prometheus-pushgateway          1/1     1            1           99s
deployment.apps/prometheus-server               1/1     1            1           99s

NAME                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/prometheus-alertmanager-d47577c4b          1         1         1       99s
replicaset.apps/prometheus-kube-state-metrics-6df5d44568   1         1         1       99s
replicaset.apps/prometheus-pushgateway-57c97d878d          1         1         1       99s
replicaset.apps/prometheus-server-559c49b4ff               1         1         1       99s

Get the Prometheus server URL by running these commands in the same shell:

export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=server" -o jsonpath=".items[0]")

Use Kubernetes port forwarding feature to access Prometheus Server.

$ kubectl --namespace monitoring port-forward $POD_NAME 9090
Forwarding from -> 9090
Forwarding from [::1]:9090 -> 9090

Access from your local machine’s browser.


Similar commands are used to access Alertmanager UI

export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=alertmanager" -o jsonpath=".items[0]")
kubectl --namespace monitoring port-forward $POD_NAME 9093

If you want to access Prometheus Server and Alertmanager with Node Ports, edit the services and set NodePort.

Next place to be is Prometheus documentation, specifically QUERYING PROMETHEUS


Gravatar Image
A systems engineer with excellent skills in systems administration, cloud computing, systems deployment, virtualization, containers, and a certified ethical hacker.