Monitor Linux Server Performance with Prometheus and Grafana in 5 minutes

By Posted on 234 views

Prometheus is an open source monitoring solution that stores all its data in a time series database. Prometheus has a multi-dimensional data-model and a powerful query language that is used to generate reports of the resources being monitored. This tutorial explains how to monitor a Linux server performance with Prometheus and Grafana.

Prometheus node exporter exports hardware and OS metrics exposed by *NIX kernels for consumption by Prometheus. This exporter is written in Go with pluggable metric collectors.

Similar Prometheus articles available on this blog are:

Setup Procedure:

  1. Install Prometheus and Grafana
  2. Install Prometheus Node Exporter on Linux servers to be monitored
  3. Configure Node Exporter
  4. Configure Prometheus server with Scrap jobs
  5. Add Dashboards to Grafana
  6. Start visualizing system metrics on Grafana

For installation of Prometheus and Grafana use:

Step 1: Add Prometheus system user

We’ll add a user account to run nod exporter service. It is safe since it doesn’t have access to the interactive shell and home directory.

sudo groupadd --system prometheus
sudo useradd -s /sbin/nologin --system -g prometheus prometheus

We added a system user called prometheus whose default group is prometheus.

Step 2: Install Prometheus Node Exporter

Next is the latest release of Prometheus Node exporter:

sudo yum -y install wget curl vim
curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest | grep browser_download_url | grep linux-amd64 |  cut -d '"' -f 4 | wget -qi -

Extract the file downloaded:

tar xvf node_exporter-*linux-amd64.tar.gz

Move the downloaded file to the /usr/local/bin directory:

cd node_exporter*/
sudo mv node_exporter /usr/local/bin/

The version installed can be confirmed using the command:

$ node_exporter  --version
node_exporter, version 1.4.0 (branch: HEAD, revision: 7da1321761b3b8dfc9e496e1a60e6a476fec6018)
  build user:       [email protected]
  build date:       20220926-12:32:56
  go version:       go1.19.1
  platform:         linux/amd64

Step 3: Configure Node Exporter systemd / Init script

Collectors are enabled by providing a --collector. flag.

Collectors that are enabled by default can be disabled by providing a --no-collector.flag.

sudo vim /etc/systemd/system/node_exporter.service

Add below content:

[Unit]
Description=Prometheus
Documentation=https://github.com/prometheus/node_exporter
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/node_exporter \
    --collector.cpu \
    --collector.diskstats \
    --collector.filesystem \
    --collector.loadavg \
    --collector.meminfo \
    --collector.filefd \
    --collector.netdev \
    --collector.stat \
    --collector.netstat \
    --collector.systemd \
    --collector.uname \
    --collector.vmstat \
    --collector.time \
    --collector.mdadm \
    --collector.zfs \
    --collector.tcpstat \
    --collector.bonding \
    --collector.hwmon \
    --collector.arp \
    --web.listen-address=:9100 \
    --web.telemetry-path="/metrics"

Start the service and enable it to start on boot:

sudo systemctl start node_exporter
sudo systemctl enable node_exporter

Step 4: Configure firewall

If you have an active firewall on your server, e.g firewalld, ufw, open port 9100

sudo ufw allow 9100

For CentOS 7 system, use firewalld:

sudo firewall-cmd --add-port=9100/tcp --permanent
sudo firewall-cmd --reload

For Init Linux system like CentOS 6, you can use daemonize to start the service in the background.

Install daemonize:

sudo yum install daemonize
sudo apt-get install daemonize

Once installed, create node_exporter init script:

sudo vim /etc/init.d/node_exporter

Add below script:

#!/bin/bash
# Author: Josphat Mutai, [email protected] , https://github.com/jmutai
# node_exporter     This shell script takes care of starting and stopping Prometheus apache exporter
#
# chkconfig: 2345 80 80
# description: Prometheus apache exporter  start script
# processname: node_exporter
# pidfile: /var/run/node_exporter.pid

# Source function library.
. /etc/rc.d/init.d/functions

RETVAL=0
PROGNAME=node_exporter
PROG=/usr/local/bin/$PROGNAME
RUNAS=prometheus
LOCKFILE=/var/lock/subsys/$PROGNAME
PIDFILE=/var/run/$PROGNAME.pid
LOGFILE=/var/log/$PROGNAME.log
DAEMON_SYSCONFIG=/etc/sysconfig/$PROGNAME

# GO CPU core Limit

#GOMAXPROCS=$(grep -c ^processor /proc/cpuinfo)
GOMAXPROCS=1

# Source config

. $DAEMON_SYSCONFIG

start() 
    if [[ -f $PIDFILE ]] > /dev/null; then
        echo "node_exporter  is already running"
        exit 0
    fi

    echo -n "Starting node_exporter  service…"
    daemonize -u $USER -p $PIDFILE -l $LOCKFILE -a -e $LOGFILE -o $LOGFILE $PROG $ARGS
    RETVAL=$?
    echo ""
    return $RETVAL


stop()  ! kill -0 $(cat "$PIDFILE"); then
        echo "Service not running"
        return 1
    fi
    echo 'Stopping service…'
    #kill -15 $(cat "$PIDFILE") && rm -f "$PIDFILE"
    killproc -p $PIDFILE -d 10 $PROG
    RETVAL=$?
    echo
    [ $RETVAL = 0 ] && rm -f $LOCKFILE $PIDFILE
    return $RETVAL


status() 
    if [ -f "$PIDFILE" ] 

# Call function
case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    restart)
        stop
        start
        ;;
    status)
        status
        ;;
    *)
        echo "Usage: $0 start"
        exit 2
esac

Create Arguments configuration file:

sudo vim /etc/sysconfig/node_exporter

Add:

"--collector.cpu \
--collector.diskstats \
--collector.filesystem \
--collector.loadavg \
--collector.meminfo \
--collector.filefd \
--collector.netdev \
--collector.stat \
--collector.netstat \
--collector.systemd \
--collector.uname \
--collector.vmstat \
--collector.time \
--collector.mdadm \
--collector.xfs \
--collector.zfs \
--collector.tcpstat \
--collector.bonding \
--collector.hwmon \
--collector.arp \
--web.listen-address=:9100

Test the script:

# /etc/init.d/node_exporter
Usage: /etc/init.d/node_exporter stop

Step 5: Start the Prometheus node exporter service

For systemd, start using:

sudo systemctl start node_exporter
sudo systemctl enable node_exporter

For Init system use:

sudo /etc/init.d/node_exporter start
sudo chkconfig node_exporter on

You can verify using:

$ sudo /etc/init.d/node_exporter status
apache exporter  service running...
Service PID: 1970

$ sudo chkconfig --list | grep node_exporter
node_exporter 0:off   1:off   2:on    3:on    4:on    5:on    6:off

$ sudo ss -tunelp | grep 9100
tcp    LISTEN     0      128      :::9100                 :::*                   users:(("node_exporter",pid=16105,fd=3)) uid:997 ino:193468 sk:ffff8a0a76f52a80 v6only:0 <->

Step 6: Add exporter job to Prometheus

The second last step is to add a job to the Prometheus server for scraping metrics. Edit /etc/prometheus/prometheus.yml

# Linux Servers
  - job_name: apache-linux-server1
    static_configs:
      - targets: ['10.1.10.20:9100']
        labels:
          alias: server1

  - job_name: apache-linux-server2
    static_configs:
      - targets: ['10.1.10.21:9100']
        labels:
          alias: server2

Restart prometheus service for scraping to start

sudo systemctl restart prometheus

Test access to port 9100 from Prometheus server

$ telnet 10.1.10.20 9100
Trying 10.1.10.20...
Connected to 10.1.10.20.
Escape character is '^]'.
^]

Step 7: Add Dashboard to Grafana

You can create your own Grafana dashboard or import from a collection of community shared dashboards. Below is a list of dashboards than has been created to show classical system metrics of your *NIX server.

Recommended: https://grafana.com/grafana/dashboards/8919

https://grafana.com/dashboards/159
https://grafana.com/dashboards/3662
https://github.com/percona/grafana-dashboards
https://github.com/rfrail3/grafana-dashboards

For demo purposes, we’ll use the first dashboard with ID 159.

Add Prometheus data source

With Prometheus data source added to Grafana, Import Apache Grafana Dashboard by navigating to Dashboard > Import. Use 159 for Grafana Dashboard ID.

prometheus-node-exporter-import-dashboard-min-768x421

Give it a descriptive name and select Prometheus data source added earlier.

prometheus-node-exporter-import-dashboard-02-min-768x285

Click “Import” button to start using the dashboard. After a few minutes, the metrics should start showing.

node-exporter-system-dashboard-min-768x355

That’s all. Feel free to customize the dashboard to fit your use case and share for others to benefit as well.

Gravatar Image
Ben Bennett - https://www.thetodaypost.net
A systems engineer with excellent skills in systems administration, cloud computing, systems deployment, virtualization, containers, and a certified ethical hacker.