Display Logs of OpenShift Nodes from oc CLI

So you have your OpenShift Cluster and you’re now on standard day 2 operations?. Checking the Logs of your Cluster Nodes is a normal operation when troubleshooting Cluster issues, e.g failed services, network & Storage issues e.t.c. In this guide, we’ll explore how one can display OpenShift Node Logs using the command line tools.

As from OpenShift Container Platform 4.0, OpenShift nodes are based on Red Hat Enterprise Linux CoreOS (RHCOS) and OKD versions uses Fedora CoreOS (FCOS). These operating systems run very few local services that would require direct access to a node to inspect their status.

Display Logs of OpenShift Nodes

Most of the system services in RHCOS / FCOS run as containers with the exceptions of the CRI-O container engine and the Kubelet, which are Systemd units.

sh-4.4# systemctl status crio
● crio.service - Open Container Initiative Daemon
   Loaded: loaded (/usr/lib/systemd/system/crio.service; disabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/crio.service.d
           └─10-default-env.conf
   Active: active (running) since Sat 2020-03-28 10:59:50 UTC; 1h 7min ago
     Docs: https://github.com/cri-o/cri-o
 Main PID: 2937 (crio)
    Tasks: 42
   Memory: 322.4M
      CPU: 5min 28.410s
   CGroup: /system.slice/crio.service
           └─2937 /usr/bin/crio --enable-metrics=true --metrics-port=9537
.....

sh-4.4# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-default-env.conf
   Active: active (running) since Sat 2020-03-28 11:00:11 UTC; 1h 7min ago
  Process: 3028 ExecStartPre=/bin/rm -f /var/lib/kubelet/cpu_manager_state (code=exited, status=0/SUCCESS)
  Process: 3026 ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)
 Main PID: 3030 (hyperkube)
    Tasks: 56 (limit: 26213)
   Memory: 241.6M
      CPU: 5min 7.547s
   CGroup: /system.slice/kubelet.service
           └─3030 /usr/bin/hyperkube kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --kubeconfig=/var/lib/kubelet/kubeco
.....

The oc adm node-logs command can be used to view these logs without requiring direct ssh to the cluster nodes. The syntax is:

oc adm node-logs [-l LABELS] [NODE...] [flags]

The command above will display all journal logs of a node. Example:

$ oc adm node-logs node01.ocp.computingpost.com

To return log entries from the specified unit(s), use the -u option:

$ oc adm node-logs  -u crio

Show logs available in a node in /var/log:

$ oc adm node-logs node01.ocp.computingpost.com --path=/
audit/
btmp
chrony/
containers/
crio/
es-containers.log.pos
fluentd/
glusterfs/
journal/
journal_pos.json
lastlog
openvswitch/
pods/
private/
samba/
sssd/
vmware-network.1.log
vmware-network.2.log
vmware-network.log
vmware-vgauthsvc.log.0
vmware-vmsvc.1.log
vmware-vmsvc.log
wtmp

Display log file from a path in a node:

oc adm node-logs node01.ocp.computingpost.com --path=/audit/audit.log

Show kubelet logs from all masters / workers:

oc adm node-logs --role master -u kubelet
oc adm node-logs --role worker -u kubelet

See what logs are available in masters/workers/infra nodes in /var/log:

oc adm node-logs --role master --path=/
oc adm node-logs --role worker --path=/
oc adm node-logs --role infra --path=/