ELK stack consists of 3 main open-source components; Elasticsearch, Logstash, and Kibana which work together to allow users to collect, analyze and visualize logs. The role of each component in the ELK stack is explained below:
- Elasticsearch – This is the core of the Elastic software. Elasticsearch is a search and analytics engine. In the ELK stack, it is used to store incoming logs from Logstash and offer the ability to search the logs in real time
- Logstash – This package collects data, transforms logs incoming from multiple sources simultaneously and sends it to storage.
- Kibana – Kibana is a graphical tool for visualizing data. It is used to generate charts and graphs to make sense of the raw data in your database.
The ELK stack can be used together with Beats for log collection. Beats can be defined as lightweight data shippers which allow multiple data sources/indices and send them to Logstash or Elasticsearch. There are several beat packages that include;
- Metricbeat – It collects metrics from systems and services including CPU, memory usage, and load, as well as other data statistics from network data and process data, before being shipped to either Logstash or Elasticsearch directly.
- Auditbeat – It is used to collect Linux audit framework data and monitor file integrity, before being shipped to either Logstash or Elasticsearch directly.
- Filebeat – Its purpose is to forward and files, usually in either .log centralise logs and or .json format.
- Heartbeat – It is used for active probing to determine whether services are available.
- Packetbeat – It supports a collection of network protocols from the application and lower-level protocols, databases and key-value stores, including HTTP, DNS, Flows, DHCPv4, MySQL and TLS. It helps identify suspicious network activities.
- Winlogbeat – It is intended to manage Windows event logs.
This guide offers an in-depth illustration of how to install and use the Elastic Stack 8 (ELK 8) on RHEL 8 / CentOS Stream 8.
1. Install Java on RHEL 8 / CentOS Stream 8
Elasticsearch requires Java 8, 11 installed on the system. If you already have Java installed, you can skip this step. Otherwise, install Java 8 using the command:
sudo yum -y install java-11-openjdk java-11-openjdk-devel
Once installed, verify the Java version.
$ java -version openjdk version "11.0.15" 2022-04-19 LTS OpenJDK Runtime Environment 18.9 (build 11.0.15+9-LTS) OpenJDK 64-Bit Server VM 18.9 (build 11.0.15+9-LTS, mixed mode, sharing)
2. Add the Elastic Stack 8.x Repositories on RHEL 8 / CentOS Stream 8
The ELK stack 8 and Beats can be downloaded from the official Elastic Stack 8.x repositories which can be added to the system using the commands below:
First, import the GPG key signing for the repository.
sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
Now add the Elastic Stack 8.x repository on RHEL 8 / CentOS Stream 8.
Once added, update the YUM package index.
sudo yum clean all sudo yum makecache
3. Install and Configure Elasticsearch 8 on RHEL 8 / CentOS 8
We will install the latest version of Elasticsearch from the added repositories using the command:
sudo yum install elasticsearch
Last metadata expiration check: 0:00:07 ago on Mon 21 Mar 2022 05:27:10 AM EDT. Dependencies resolved. ================================================================================ Package Architecture Version Repository Size ================================================================================ Installing: elasticsearch x86_64 8.1.0-1 elasticsearch-8.x 492 M Transaction Summary ================================================================================ Install 1 Package Total download size: 492 M Installed size: 1.0 G Is this ok [y/N]: y
Configure Elasticsearch by editing the below YAML file.
sudo vi /etc/elasticsearch/elasticsearch.yml
In the file, make the below adjustments
# Path to directory where to store the data (separate multiple locations by comma): # path.data: /var/lib/elasticsearch # # Path to log files: # path.logs: /var/log/elasticsearch # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 0.0.0.0 . . . # -------------------------------------------------------------------------------- # Enable security features xpack.security.enabled: false xpack.security.enrollment.enabled: false # Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents xpack.security.http.ssl: enabled: false keystore.path: certs/http.p12 # Enable encryption and mutual authentication between cluster nodes xpack.security.transport.ssl: enabled: false
If you set a custom data path for Elasticsearch, you need to disable SELinux or set it in permissive mode for the path to be accessible.
sudo setenforce 0
Configure the JVM options by setting the memory limits:
$ sudo vi /etc/elasticsearch/jvm.options -Xms1g -Xmx1g
If the system has low memory, configure it to use the least memory:
Start and enable Elasticsearch 8 using the command:
sudo systemctl start elasticsearch sudo systemctl enable elasticsearch
Verify if Elasticsearch is responding to queries:
$ curl -X GET localhost:9200 "name" : "localhost.localdomain", "cluster_name" : "elasticsearch", "cluster_uuid" : "_yNoaSr4SeShhkRNDkDC6A", "version" : "number" : "8.1.0", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "3700f7679f7d95e36da0b43762189bab189bc53a", "build_date" : "2022-03-03T14:20:00.690422633Z", "build_snapshot" : false, "lucene_version" : "9.0.0", "minimum_wire_compatibility_version" : "7.17.0", "minimum_index_compatibility_version" : "7.0.0" , "tagline" : "You Know, for Search"
4. Install and Configure Logstash on RHEL 8 / CentOS Stream 8
Install Logstash on RHEL 8 / CentOS Stream 8 using the command:
sudo yum -y install logstash
Once installed, you need to configure Logstash by creating a config at /etc/logstash/conf.d/. The file will have 3 sections i.e the input, filter, and output. All the 3 sections can exist in a single or in 3 different files.
sudo vi /etc/logstash/conf.d/beats.conf
In this guide, we will have all the 3 sections in a single file as below:
input beats port => 5044 filter if [type] == "syslog" grok match => "message" => "%SYSLOGLINE" date match => [ "timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] output elasticsearch hosts => ["192.168.205.8:9200"] index => "%[@metadata][beat]-%+YYYY.MM.dd"
Start and enable Logstash.
sudo systemctl start logstash sudo systemctl enable logstash
5. Install and Configure Kibana on RHEL 8 / CentOS 8
Now we want to install Kibana, the visualization tool in the ELK stack. Install Kibana using the command:
sudo yum -y install kibana
Once installed, configure Kibana.
sudo vi /etc/kibana/kibana.yml
In the opened file, make the below adjustments:
# line 11 : uncomment and change (listen all) server.host: "0.0.0.0" # line 32 : uncomment and change (specify own hostname) server.name: "node1" # line 43 : uncomment and change if you need # set if elasticsearch and Kibana are running on different Host elasticsearch.hosts: ["http://192.168.205.8:9200"]
Start and enable Kibana.
sudo systemctl enable --now kibana
Allow ports 5601 and 5044 through the firewall.
sudo firewall-cmd --add-port=5601/tcp --permanent sudo firewall-cmd --add-port=5044/tcp --permanent sudo firewall-cmd --reload
6. Install and Configure Filebeat on RHEL 8 / CentOS 8
At this point, the Kibana dashboard is available on port 5601. We want to install and configure Filebeat to help collect logs and send them to Logstash which transforms them into a format easily understood by Elasticsearch. Filebeat can be installed on the same or client machine from which you want to collect the logs.
Install Filebeat on RHEL 8 / CentOS Stream 8 using the command:
sudo dnf -y install filebeat
Once installed, proceed and configure it to send logs to Logstash as below:
sudo vi /etc/filebeat/filebeat.yml
In the opened file, make the below configurations.
#-------------------------- Elasticsearch output ------------------------------ #output.elasticsearch: # Array of hosts to connect to. #hosts: ["localhost:9200"] ..... #----------------------------- Logstash output -------------------------------- output.logstash: # The Logstash hosts hosts: ["192.168.205.8:5044"] .... filebeat.inputs: # Each - is an input. Most options can be set at the input level, so # you can use different inputs for various configurations. # Below are the input specific configurations. - type: log # Change to true to enable this input configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /var/log/messages #- c:\programdata\elasticsearch\logs\*
Start and enable Filebeat.
sudo systemctl enable --now filebeat
7. Access the Kibana Web Interface.
The Kibana web interface can be accessed using the URL http://IP_Address:5601.
Now enable the Filebeat module on the client machine. Begin by listing modules:
$ sudo filebeat modules list Enabled: Disabled: activemq apache auditd aws awsfargate azure barracuda bluecoat cef checkpoint cisco coredns ....
Now enable the desired module; Logstash as below:
$ sudo filebeat modules enable logstash Enabled logstash
Once enabled, load the index template to Elasticsearch and dashboards to Kibana as below.
$ sudo filebeat setup -E output.logstash.enabled=false -E output.elasticsearch.hosts=['192.168.205.8:9200'] -E setup.kibana.host=192.168.205.8:5601 Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling. Index setup finished. Loading dashboards (Kibana must be running and reachable) Loaded dashboards Loaded Ingest pipelines
Now on the Kibana interface under discover, you will have Filebeat as below.
You can also visualize the data using desired chart/graph.
At this point, we are safe to assume that the Elastic Stack 8 (ELK 8) setup on RHEL 8 / CentOS Stream 8 is working perfectly. We can now collect logs and send them to Logstash or Elasticsearch. I hope this was significant.