Install Elastic Stack 8 (ELK) on Amazon Linux 2

Import the GPG key.

sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Update the system cache.

sudo yum clean all
sudo yum makecache

Install Elasticsearch on Amazon Linux 2.

sudo yum -y install elasticsearch

Verify that Elasticearch has been installed successfully:

[[email protected] ~]# rpm -qi elasticsearch
Name        : elasticsearch
Epoch       : 0
Version     : 8.0.1
Release     : 1
Architecture: x86_64
Install Date: Thu Mar  3 22:12:37 2022
Group       : Application/Internet
Size        : 1100107680
License     : Elastic License
Signature   : RSA/SHA512, Thu Feb 24 18:00:40 2022, Key ID d27d666cd88e42b4
Source RPM  : elasticsearch-8.0.1-1-src.rpm
Build Date  : Thu Feb 24 14:03:12 2022
Build Host  : packer-virtualbox-iso-1636998457
Relocations : /usr
Packager    : Elasticsearch
Vendor      : Elasticsearch
URL         : https://www.elastic.co/

Configure Elasticsearch on Amazon Linux 2

After the installation, you may need to configure Elasticsearch to add your cluster name and the bind-address. Edit the file /etc/elasticsearch/elasticsearch.yml

$ sudo vi /etc/elasticsearch/elasticsearch.yml
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: elk-cluster
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 0.0.0.0
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.

You can also modify Java memory options for Elasticsearch by editing the file /etc/elasticsearch/jvm.options

$ sudo vi /etc/elasticsearch/jvm.options
-Xms1g
-Xmx1g

The lines above set the maximum memory size to 1GB.

Start and enable Elasticsearch service.

sudo systemctl enable --now elasticsearch.service 

Verify service status

[[email protected] ~]# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2021-08-15 13:05:05 UTC; 16s ago
     Docs: https://www.elastic.co
 Main PID: 2637 (java)
   CGroup: /system.slice/elasticsearch.service
           ├─2637 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encod...
           └─2818 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Aug 15 13:04:39 amazon-linux systemd[1]: Starting Elasticsearch...
Aug 15 13:05:05 amazon-linux systemd[1]: Started Elasticsearch.
[[email protected] ~]# 

Verify that Elasticsearch is working:

 $ curl http://127.0.0.1:9200 

  "name" : "amazon-linux",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "DzAF-DclTYqC9zG9S5uWow",
  "version" : 
    "number" : "7.14.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "dd5a0a2acaa2045ff9624f3729fc8a6f40835aa1",
    "build_date" : "2021-07-29T20:49:32.864135063Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  ,
  "tagline" : "You Know, for Search"

Test Elasticsearch indexing:

$ curl -X GET "localhost:9200"

  "name" : "amazon-linux",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "DzAF-DclTYqC9zG9S5uWow",
  "version" : 
    "number" : "7.14.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "dd5a0a2acaa2045ff9624f3729fc8a6f40835aa1",
    "build_date" : "2021-07-29T20:49:32.864135063Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  ,
  "tagline" : "You Know, for Search"

Install Logstash on Amazon Linux 2

After a successful installation and configuration of Elasticsearch on Amazon Linux 2, we now proceed to the next element, which is Logstash.

Install Logstash with the command below:

sudo yum install logstash -y

Edit the Logstash configuration file to add the input and output parameters.

$ sudo vi /etc/logstash/conf.d/logstash.conf

input 

  beats 

    port => 5044

  


output 

  elasticsearch 

    hosts => ["localhost:9200"]

    manage_template => false

    index => "%[@metadata][beat]-%[@metadata][version]-%+YYYY.MM.dd"

  

Start and enable Logstash

systemctl enable --now logstash

Install and Configure Kibana on Amazon Linux 2

The next item to be installed will be Kibana.

Kibana exists in the ELK repo that we had configured earlier, we shall therefore proceed to install the package directly.

sudo yum -y install kibana

Configure Kibana by adding the port binding parameters:

$ sudo vim /etc/kibana/kibana.yml
server.host: "0.0.0.0"
elasticsearch.url: "http://localhost:9200"

Start and enable Kibana service:

systemctl enable --now kibana

Verify that the service is up.

$ systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2021-08-15 13:15:04 UTC; 7s ago
     Docs: https://www.elastic.co
 Main PID: 2941 (node)
   CGroup: /system.slice/kibana.service
           ├─2941 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist --logging.dest="/var/log/kibana/kibana.log" --pid.file="/run/kibana/kibana.pid"
           └─2955 /usr/share/kibana/node/bin/node --preserve-symlinks-main --preserve-symlinks /usr/share/kibana/src/cli/dist --logging.dest="/var/log/kibana/kibana.log" --pid.file="/run/kibana/kibana.pid"

Aug 15 13:15:04 amazon-linux systemd[1]: Started Kibana.

Allow Kibana through the firewall:

sudo firewall-cmd --permanent --add-port=5601/tcp
sudo firewall-cmd --reload

Access the Kibana dashboard by http://server-ip:5601.

You can now start adding your data and shipping logs using beats such as Filebeat, Metricbeat etc.

Install Filebeat on Amazon Linux 2

Filebeat is a beat that is used to send log files to the ELK cluster for different applications and services.

Run the command below to install Filebeat on Amazon Linux 2.

sudo yum install filebeat

Enable modules for Filebeat. This enables the applications that will ship their logs to Elasticsearch. To check the available modules, run the command below:

sudo filebeat modules list

Enable a module, such as the Nginx module:

sudo filebeat modules enable system

Run the Filebeat setup to initialize the Filebeat process

$ sudo filebeat setup
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Setting up ML using setup --machine-learning is going to be removed in 8.0.0. Please use the ML app instead.
See more: https://www.elastic.co/guide/en/machine-learning/current/index.html
Loaded machine learning job configurations
Loaded Ingest pipelines

Start Filebeat service

sudo service filebeat start

Go to the Kibana dashboard and verify that you can see the metrics for the enabled module.

That’s all for this setup.

Wrap Up

We have successfully set up ELK stack on Amazon Linux 2. You can use this stack to monitor logs for your distributed systems. This helps in improving business productivity by allowing system admins to plan for their systems in terms of resources, security, etc.