As a system administrator, chances are that there are some tasks that you do repetitively every single day that takes up a lot of your time that could have been invested elsewhere. What if you could solve problems once and then automate your solutions going forward? That is what Ansible is good at and once you get to see its benefits, you will never look back again. Ansible is a simple, yet powerful IT automation engine that thousands of companies are using to drive complexity out of their environments and accelerate DevOps initiatives. In this article we will perform an installation of Ansible AWX on CentOS 8 / Rocky Linux 8 Server.
Be it the deployment of applications, routine maintenance of your servers, Configuration Management, Continuous Delivery, Orchestration or any repetitive work that you can describe, Ansible can handle it for you. To add beauty to this beast, AWX (Ansible Web eXecutable) provides a web-based user interface, REST API, and task engine built on top of Ansible. The AWX Project is an open source community project, sponsored by Red Hat, that enables users to better control their use of Ansible project in IT environments. AWX is the upstream project from which the Red Hat Ansible Tower offering is ultimately derived.
“Don’t wait. The time will never be just right.”
Setup minimum requirements
Before you can run a deployment, you’ll need the following installed in your local environment
- Kubernetes Cluster / Docker
- User with sudo access
- CentOS 8 / Rocky Linux 8 server
- At least 8GB of RAM – More is better if available
- 4vcpus – Minimum CPU but add more if available
- 25GB minimum disk space
We should be now ready to roll up our sleeves, put on our boots and get to work.
Step 1: Update your system
The first step is performing a system update.
sudo dnf -y update
Disable Firewalld. This is recommended by K3s.
sudo systemctl disable firewalld --now
Once the update is successful perform a system reboot
Step 2: Install K3s Kubernetes Distribution
AWX is supported and can only be run as a containerized application using Docker images deployed to either an OpenShift cluster, a Kubernetes cluster, or docker-compose. We shall use K3s Kubernetes setup to run AWX on CentOS 8 / Rocky Linux 8.
Put SELinux in permissive mode:
sudo setenforce 0 sudo sed -i 's/^SELINUX=.*/SELINUX=permissive/g' /etc/selinux/config cat /etc/selinux/config | grep SELINUX=
Install k3s by running the commands below:
curl -sfL https://get.k3s.io | sudo bash - sudo chmod 644 /etc/rancher/k3s/k3s.yaml
Check k3s service to confirm it is running and working:
$ systemctl status k3s.service ● k3s.service - Lightweight Kubernetes Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled) Active: active (running) since Sun 2021-07-04 22:28:01 UTC; 10s ago Docs: https://k3s.io Process: 6273 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS) Process: 6265 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS) Process: 6262 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS) Main PID: 6277 (k3s-server) Tasks: 30 Memory: 668.9M CGroup: /system.slice/k3s.service ├─6277 /usr/local/bin/k3s server └─6299 containerd ....
As root user do a validation on use of kubectl Kubernetes management tool:
$ sudo su - # kubectl get nodes NAME STATUS ROLES AGE VERSION centos.hirebestengineers.com Ready control-plane,master 111s v1.23.6+k3s1
You can also confirm Kubernetes version deployed using the following command:
# kubectl version --short Client Version: v1.23.6+k3s1 Server Version: v1.23.6+k3s1
The K3s service will be configured to automatically restart after node reboots or if the process crashes or is killed.
Step 3: Deploy AWX Operator on Kubernetes
This Kubernetes Operator has to be deployed for the management of one or more AWX instances in any namespace.
Install git and make tools:
sudo yum -y install git make
Clone operator deployment code:
# git clone https://github.com/ansible/awx-operator.git Cloning into 'awx-operator'... remote: Enumerating objects: 5626, done. remote: Counting objects: 100% (2840/2840), done. remote: Compressing objects: 100% (1025/1025), done. remote: Total 5626 (delta 1910), reused 2434 (delta 1716), pack-reused 2786 Receiving objects: 100% (5626/5626), 1.38 MiB | 12.01 MiB/s, done. Resolving deltas: 100% (3191/3191), done.
Create namespace where operator will be deployed. I’ll name mine awx:
export NAMESPACE=awx kubectl create ns $NAMESPACE
Set current context to value set in NAMESPACE variable:
# kubectl config set-context --current --namespace=$NAMESPACE Context "default" modified.
Switch to awx-operator directory:
Save the latest version from AWX Operator releases as RELEASE_TAG variable then checkout to the branch using git.
sudo yum -y install jq RELEASE_TAG=`curl -s https://api.github.com/repos/ansible/awx-operator/releases/latest | grep tag_name | cut -d '"' -f 4` echo $RELEASE_TAG
Deploy AWX Operator into your cluster:
git checkout $RELEASE_TAG export NAMESPACE=awx make deploy
Command execution terminal output:
namespace/awx configured customresourcedefinition.apiextensions.k8s.io/awxbackups.awx.ansible.com created customresourcedefinition.apiextensions.k8s.io/awxrestores.awx.ansible.com created customresourcedefinition.apiextensions.k8s.io/awxs.awx.ansible.com created serviceaccount/awx-operator-controller-manager created role.rbac.authorization.k8s.io/awx-operator-awx-manager-role created role.rbac.authorization.k8s.io/awx-operator-leader-election-role created clusterrole.rbac.authorization.k8s.io/awx-operator-metrics-reader created clusterrole.rbac.authorization.k8s.io/awx-operator-proxy-role created rolebinding.rbac.authorization.k8s.io/awx-operator-awx-manager-rolebinding created rolebinding.rbac.authorization.k8s.io/awx-operator-leader-election-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/awx-operator-proxy-rolebinding created configmap/awx-operator-awx-manager-config created service/awx-operator-controller-manager-metrics-service created deployment.apps/awx-operator-controller-manager created
Wait a few minutes and awx-operator should be running:
# kubectl get pods -n awx NAME READY STATUS RESTARTS AGE awx-operator-controller-manager-68d787cfbd-z75n4 2/2 Running 0 48s
Uninstalling AWX Operator – Don’t run, only for reference🙂
You can always remove the operator and all associated CRDs by running the command below:
# export NAMESPACE=awx # make undeploy /root/awx-operator/bin/kustomize build config/default | kubectl delete -f - namespace "awx" deleted customresourcedefinition.apiextensions.k8s.io "awxbackups.awx.ansible.com" deleted customresourcedefinition.apiextensions.k8s.io "awxrestores.awx.ansible.com" deleted customresourcedefinition.apiextensions.k8s.io "awxs.awx.ansible.com" deleted serviceaccount "awx-operator-controller-manager" deleted role.rbac.authorization.k8s.io "awx-operator-leader-election-role" deleted role.rbac.authorization.k8s.io "awx-operator-manager-role" deleted clusterrole.rbac.authorization.k8s.io "awx-operator-metrics-reader" deleted clusterrole.rbac.authorization.k8s.io "awx-operator-proxy-role" deleted rolebinding.rbac.authorization.k8s.io "awx-operator-leader-election-rolebinding" deleted rolebinding.rbac.authorization.k8s.io "awx-operator-manager-rolebinding" deleted clusterrolebinding.rbac.authorization.k8s.io "awx-operator-proxy-rolebinding" deleted configmap "awx-operator-manager-config" deleted service "awx-operator-controller-manager-metrics-service" deleted deployment.apps "awx-operator-controller-manager" deleted
Step 4: Install Ansible AWX on CentOS 8 / Rocky Linux 8
Now that we have the operator pod running we are ready to initiate installation of Ansible AWX on CentOS 8 / Rocky Linux 8. But first we’ll need to create a PVC for public and static web data.
Create a file named public-static-pvc.yaml:
Input below contents in the file:
--- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: public-static-data-pvc spec: accessModes: - ReadWriteOnce storageClassName: local-path resources: requests: storage: 5Gi
Apply configuration manifest:
# kubectl apply -f public-static-pvc.yaml -n awx persistentvolumeclaim/public-static-data-pvc created
PVC won’t be bound until the pod that uses it is created.
# kubectl get pvc -n awx NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE public-static-data-pvc Pending local-path 43s
Create AWX deployment file:
Paste below contents to the file created.
--- apiVersion: awx.ansible.com/v1beta1 kind: AWX metadata: name: awx spec: service_type: nodeport projects_persistence: true projects_storage_access_mode: ReadWriteOnce web_extra_volume_mounts: | - name: static-data mountPath: /var/lib/projects extra_volumes: | - name: static-data persistentVolumeClaim: claimName: public-static-data-pvc
Install AWX on CentOS 8 / Rocky Linux 8:
# kubectl apply -f awx-instance-deployment.yml -n awx awx.awx.ansible.com/awx created
After few minutes check pods creation status:
# watch kubectl get pods -l "app.kubernetes.io/managed-by=awx-operator" -n awx NAME READY STATUS RESTARTS AGE awx-postgres-0 1/1 Running 0 2m58s awx-75698588d6-qz2gf 4/4 Running 0 2m42s
You can track the installation process at the operator pod logs:
kubectl logs -f deployments/awx-operator-controller-manager -c awx-manager
Extra PVCs are created automatically:
# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE postgres-awx-postgres-0 Bound pvc-34a25045-486c-42a8-9763-d14a7bb3e9e8 8Gi RWO local-path 72s public-static-data-pvc Bound pvc-3484e513-8d00-482c-bdce-6e77820f237e 1Gi RWO local-path 5m13s awx-projects-claim Bound pvc-e56ab471-97f1-455b-9d51-ba05b7d9982b 8Gi RWO local-path 60s
Fixing the error “mkdir: cannot create directory ‘/var/lib/postgresql/data’: Permission denied”
If you see the error message from postgres pod logs:
# kubectl logs awx-postgres-0 mkdir: cannot create directory ‘/var/lib/postgresql/data’: Permission denied
It means the Postgres pod cannot write to the persistent volume directory inside/var/lib/rancher/k3s/storage/:
# ls -lh /var/lib/rancher/k3s/storage/ | grep awx-postgres-0 total 0 drwx------. 3 root root 18 Aug 3 14:04 pvc-8110b494-d9ed-450a-94c0-b9dfd2bd73f7_default_postgres-awx-postgres-0
Try setting the directory mode to 777
# chmod -R 777 /var/lib/rancher/k3s/storage/* # kubectl delete pods -l "app.kubernetes.io/managed-by=awx-operator" -n awx pod "awx-75698588d6-x79g2" deleted pod "awx-postgres-0" deleted
The Postgres container pod should come up in few seconds:
# kubectl get pods -n awx NAME READY STATUS RESTARTS AGE awx-operator-545497f7d5-bqlcs 1/1 Running 0 65m awx-postgres-0 1/1 Running 4 8m22s awx-75698588d6-7kg9j 4/4 Running 0 8m10s
Checking AWX Container’s logs
The awx-xxx-yyy pod will have four containers, namely:
As can be seen from below command output:
# kubectl get deploy NAME READY UP-TO-DATE AVAILABLE AGE awx-operator-controller-manager 1/1 1 1 9m58s awx 1/1 1 1 7m47s # kubectl -n awx logs deploy/awx error: a container name must be specified for pod awx-66596c8fcb-s28tw, choose one of: [redis awx-web awx-task awx-ee] or one of the init containers: [database-check init]
You’ll need to provide container name after the pod:
kubectl -n awx logs deploy/awx -c redis kubectl -n awx logs deploy/awx -c awx-web kubectl -n awx logs deploy/awx -c awx-task kubectl -n awx logs deploy/awx -c awx-ee
Access AWX Container’s Shell
Here is how to access each container’s shell:
kubectl exec -ti deploy/awx -c awx-task -- /bin/bash kubectl exec -ti deploy/awx -c awx-web -- /bin/bash kubectl exec -ti deploy/awx -c awx-ee -- /bin/bash kubectl exec -ti deploy/awx -c redis -- /bin/bash
Upgrading AWX Instance
For upgrade process refer to our guide in the link below:
Step 4: Access AWX Web Interface
Get the AWX Web service port:
# kubectl get service -n awx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE awx-postgres ClusterIP None
5432/TCP 5m17s awx-service NodePort 10.43.54.203 80:30080/TCP 5m7s
From the output we can confirm service node port is 30080.
To have access to AWX web console, point your browser to your Ansible’s AWX server IP:
You should be welcomed to a Login page well illustrated below.
The login username is admin
Obtain admin user password by decoding the secret with the password value:
kubectl -n awx get secret awx-admin-password -o jsonpath=".data.password" | base64 --decode
Better output format:
kubectl -n awx get secret awx-admin-password -o go-template='range $k,$v := .dataprintf "%s: " $kif not $v$velse$v end"\n"end'
Login with the admin username and decoded password from above commands:
Once the authentication is successful, you’ll get to AWX administration dashboard. Therein, there is a lot of stuff to do and we shall cover just a few. Along the left side of the Dashboard is the navigation menu, where you can quickly navigate to your Projects, Inventories, Job Templates, and Jobs
Step 5: Create User and Team
AWX provides a default user called admin that you can use to do your tests. But if you are need a different user, you can create another one.
To add a new user apart from the admin one you find by default, click on Users tab found on the left menu. A new page will be displayed where you can add a new user. Click on the green + icon then fill in the details of the new user to be added.
After you are done, click “Save“.
The same applies for creating a new Team. Click on “Teams” tab found on the left menu. A new page will be displayed where you can add a new team. Click on the green + icon then fill in the details of the new team to be added.
After you are done, click “Save“.
Step 6: Create an Organization
An organization is a logical collection of users, teams, projects, and inventories. It is the highest level object in the AWX object hierarchy. From the left navigation bar, click the Organizations icon. You will find that a default organization has been automatically created and is available to all users of Ansible AWX.
It can be used as is or edited as needed. Let us edit it by changing its name and adding users, permissions and more.
Click on the “Users” tab and add a new user. I had not created a new user before, so admin default user appears. But you can create other users as shared in Step 5.
If you had other users, they would be populated therein
Clicking on Users displays all the Users associated with this Organization. A User is someone with access to Ansible AWX with associated roles and Credentials. Adding a user to an organization adds them as a member only, specifying a role for the user can be done in the the Permissions tab, as shown in the example below:
Step 7: Create credentials
In order for Ansible to log in and execute tasks, it will need credentials to access the remote hosts. AWX provides this feature for us. Click on “Credentials” on the left menu and then click the green + to add a new server credential.
Fill in the name, description, organization, username, password and the type of credential which is a long list. We shall choose machine since we intend to configure a single host in this example. You can add your ssh keys in case you prefer key-based authentication mechanism. Also add privilege escalation below to sudo.
Click “Save” once done.
Step 8: Create a new Inventory and add it to the Organization
An inventory is a collection of hosts managed by Ansible. Inventories are assigned to organizations, while permissions to launch playbooks against inventories are controlled at the user and/or team level.
To create and review existing inventories click the Inventories icon from the left navigation bar. To create a new inventory, click the add (+) button and select Inventory from the drop-down menu list.
Enter the name and Organization that this Inventory will belong to. Click “Save” so that the other tabs can be activated.
For this example, we are going to add one host but know that you can create a group that contains one or more hosts you would wish to execute something on them simultaneously. Click on “Hosts” tab and click on + to add a new host.
On the new page loaded, add an IP or resolvable hostname and a description. We are going to install nginx on the given host in this example thus the intuitive name.
Click “Save” once done and the new Inventory should be created.
The concept of Groups and Hosts
Note that inventories are divided into groups and hosts. A group might represent a particular environment (e.g. “Datacenter 1” or “Testing Bed”), a server type (e.g. “Web Servers” or “DB Servers”), or any other representation of your environment.
Step 9: Setting up a Project
A Project is a logical collection of Ansible playbooks, represented in Ansible AWX. You can manage playbooks and playbook directories by either placing them manually under the Project Base Path on your Ansible AWX server, or by placing your playbooks into a source code management (SCM) system supported by Ansible AWX, including Git, Subversion, and Mercurial.
To create a new project, follow the same procedure as the rest we have seen thus far. Click on the “Projects” tab on the left menu and then click on add(+) to create a new project.
Once there, fill the details to suit your needs. On the Source Control Manager (SCM) type, you can create a git repo that has all of your playbooks or create a local folder on your server as advised when you choose manual under SCM type. I preferred adding a git repo in this example. Note that we are still referring to the Organization of your choice in case you have several.
Enter all the details then click “Save“. The file created on the git repo (nginx.yml) has the following:
--- - hosts: all gather_facts: true become: true become_user: root tasks: - apt: name: nginx when: ansible_os_family == "Debian"
Attached screenshot below:
Step 10: Setting up a template and launching it
Thus far, we have done a lot and what remains is putting it all together into a template. A job template combines an Ansible playbook from a project and the settings required to launch it. Create a new job template by clicking on “Templates” tab on the left navigation menu. Click on he green add (+) to add a new template just like the others. When the window opens, it is just a matter of picking the Inventory, Project, Credential and Playbook we had already configured in the previous steps. Give your template a name and an optional description as well.
When done scroll down and hit “Save“. Once you save, you can “Launch” it as a job immediately.
Once you hit Launch you will be redirected to the “Jobs” page where you will see it executing live with output being displayed as it happens.
If everything goes well, the playbook should be run successfully by Ansible behind the scenes. All errors encountered along the way will be displayed on the right side.
Login to the server being tested on and check if nginx was installed successfully
Amazing stuff, Ansible AWX installation is officially setup and running properly.
More about Ansible Tower | AWX can be found on RedHat’s Ansible Tower Quick Start Guide.
Ansible can change the way you handle your day to day administration tasks and it can offload a lot of manual work from your hands and gift you with ample time for other tasks. Leverage this technology as your innovation engine, and deliver your applications faster and win big.