After setting up your Kubernetes cluster in the cloud or on an on-premise environment, there will be a need to check what is going on inside your warehouse in an easy and flexible fashion. One of the best ways to do that is to investigate logs whenever you need to fix or know what happened at a particular time. While it is possible to log into your cluster and check the logs of your pods or your hosts, it suddenly becomes tedious to check each pod’s logs individually especially if you have many pods in your k8s. To make it easier for you to check the status of your cluster on one platform, we are going to deploy Elasticsearch and Kibana on an external server then ship logs from your cluster to Elasticsearch using Elastic’s beats (Filebeat, Metricbeat etc). If you already have an ELK Stack already running, then the better.
The diagram below shows the architecture we shall accomplish in this guide. It is essentially a 3 node Kubernetes cluster and one Elasticsearch and Kibana server which will be receiving logs from the cluster via Filebeat and Metricbeat log collectors.
First, we shall need an Elasticsearch server with with Kibana installed as well. You can leave out Logstash but incase you need further filtering of your logs, then you can have it installed. Follow the guides below to install Elasticsearch and Kibana:
On your Elasticsearch host, make sure it can be reached from outside. You can edit the following parts in your config
$ sudo vim /etc/elasticsearch/elasticsearch.yml # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 0.0.0.0 # # Set a custom port for HTTP: # http.port: 9200 #
Then allow the port on your firewall
sudo firewall-cmd --permanent --add-port=9200/tcp sudo firewall-cmd --reload
Secondly, you must have a Kubernetes cluster since it is the place we are concerned about fetching logs from. We have a couple of guides that will help you set one up in case you need to bootstrap one quickly. They are shared below:
After you are ready, we can proceed to install Filebeat and Metricbeat pods in our cluster to start collecting and sending the logs to ELK. Make sure you are able to run kubectl command in your Kubernetes cluster.
Step 1: Download Sample Filebeat and Metricbeat files
Log into your Kubernetes master node and run the command below to fetch Filebeat and Metricbeat yaml files provided by Elastic.
cd ~ curl -L -O https://raw.githubusercontent.com/elastic/beats/7.9/deploy/kubernetes/filebeat-kubernetes.yaml curl -L -O https://raw.githubusercontent.com/elastic/beats/7.9/deploy/kubernetes/metricbeat-kubernetes.yaml
Step 2: Edit the files to befit your environment
In both files, we only need to change a few things. Under the ConfigMap, you will find elasticseach output as shown below. Change the ip (192.168.10.123) and port (9200) to that of your Elasticsearch server.
output.elasticsearch: hosts: ['$ELASTICSEARCH_HOST:192.168.10.123:$ELASTICSEARCH_PORT:9200'] #username: $ELASTICSEARCH_USERNAME #password: $ELASTICSEARCH_PASSWORD
Under DaemonSet within the same file, you will find the following configuration. Note that we are showing the areas to change. Edit the IP (192.168.10.123) and port (9200) to match that of your Elastcsearch server as well. If you have configured username and password for your Elasticsearch, you are free to add them on the commented sections shown.
env: - name: ELASTICSEARCH_HOST value: "192.168.10.123" - name: ELASTICSEARCH_PORT value: "9200" #- name: ELASTICSEARCH_USERNAME # value: elastic #- name: ELASTICSEARCH_PASSWORD # value: changeme - name: ELASTIC_CLOUD_ID
It should be noted that if you would wish to deploy your Filebeat and Metricbeat resources on another namespace, simply edit all instances of “kube-system” to the one of your choice.
Under Deployment, you can change the version of Filebeat and Metricbeat to be deployed by editing the image (docker.elastic.co/beats/metricbeat:7.9.0) seen on the snippet below. I am going to use version 7.9.0.
###For Metricbeat#### spec: serviceAccountName: metricbeat hostNetwork: true dnsPolicy: ClusterFirstWithHostNet containers: - name: metricbeat image: docker.elastic.co/beats/metricbeat:7.9.0
Do the same for the filebeat yml file if you would wish to change its version as well.
###For Filebeat#### spec: serviceAccountName: metricbeat hostNetwork: true dnsPolicy: ClusterFirstWithHostNet containers: - name: metricbeat image: docker.elastic.co/beats/filebeat:7.9.0
Important thing to note
If you would wish to deploy the beats on the master node, we will have to add tolerations. An example for metricbeat is shown below. That is not the whole DaemonSet config. Just the part we are interested in. You can leave the other parts intact. Add the toleration as shown on the configuration below under spec. The same can apply on filebeat configuration depending on your needs.
apiVersion: apps/v1 kind: DaemonSet metadata: name: metricbeat namespace: kube-system labels: k8s-app: metricbeat spec: selector: matchLabels: k8s-app: metricbeat template: metadata: labels: k8s-app: metricbeat spec: ###PART TO EDIT### # This toleration is to have the daemonset runnable on master nodes # Remove it if your masters can't run pods tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule ####END OF EDIT### serviceAccountName: metricbeat terminationGracePeriodSeconds: 30 hostNetwork: true dnsPolicy: ClusterFirstWithHostNet containers: - name: metricbeat image: docker.elastic.co/beats/metricbeat:7.9.0 args: [ "-c", "/etc/metricbeat.yml", "-e",
Step 4: Deploying to Kubernetes
After we have done all of our edits and our Elasticsearch is well reachable from your Kubernetes cluster, It is time to deploy our beats. Login to your master node and run the commands below:
kubectl apply -f metricbeat-kubernetes.yaml kubectl apply -f filebeat-kubernetes.yaml
Confirm that the pods are deployed and running successfully after some time.
$ kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-c9784d67d-k85hf 1/1 Running 5 11d calico-node-brjnk 1/1 Running 7 10d calico-node-nx869 1/1 Running 1 10d calico-node-whlzf 1/1 Running 6 11d coredns-f9fd979d6-6vztd 1/1 Running 5 11d coredns-f9fd979d6-8gz4l 1/1 Running 5 11d etcd-kmaster.diab.mfs.co.ke 1/1 Running 5 11d filebeat-hlzhc 1/1 Running 7 7d23h <== filebeat-mcs67 1/1 Running 1 7d23h <== kube-apiserver-kmaster.diab.mfs.co.ke 1/1 Running 5 11d kube-controller-manager-kmaster.diab.mfs.co.ke 1/1 Running 5 11d kube-proxy-nlrbv 1/1 Running 5 11d kube-proxy-zdcbg 1/1 Running 1 10d kube-proxy-zvf6c 1/1 Running 7 10d kube-scheduler-kmaster.diab.mfs.co.ke 1/1 Running 5 11d metricbeat-5fw98 1/1 Running 7 8d <== metricbeat-5zw9b 1/1 Running 0 8d <== metricbeat-jbppx 1/1 Running 1 8d <==
Step 5: Create Index on Kibana
Once our Pods begin running, they will immediately send an index pattern to Elasticsearch together with the logs. Login to your Kibana and Click “Stack Management” > “Index Management” and you should be able to see your indexes.
Click on “Index Management“
And there are our indexes.
To create an index pattern, click on “Index Patterns” then hit “Create Index Pattern“.
On the next page, type in the name of the index pattern that matches either filebeat or metricbeat and they should show up as already matched.
Create the pattern then click “Next Step“
Choose “@timestamp” on the drop-down then “Create Index Pattern“
Step 6: Discover your Data
After the index pattern has been created, click on “Discover”,
Then choose the index pattern we created.
Now we have lightweight beats that fetch logs and metrics from your Kubernetes cluster and ship them to an external Elasticsearch for indexing and flexible searching.