One of the many advantages that Kubernetes has to offer is auto-scaling. This enables the automatic provisioning of resources based on traffic/demand. As the traffic demand increases more resources are provisioned and vice versa. Kubernetes offers three types of scaling:
- Vertical Pod autoscaler: Scaling that increases/decreases the CPU or memory allocated to Pods.
- Horizontal Pod autoscaler: Increase/Decreases the number of pods per service in response to traffic/demand.
- Cluster autoscaler: Scales your cluster nodes based on the number of pending pods.
Running Kubernetes on AWS (EKS) offers the option to scale your cluster nodes based on metrics from your cluster resource utilization. This article guides the reader on configuring auto-scaling based on cluster metrics. We will use the Cluster Node Memory utilization metric to scale our nodes.
N/B: This is only possible on an EKS cluster with a managed node group (Provisioned Worker Nodes) and not an EKS Fargate cluster. With Fargate there is no need to provision instances for your cluster because it provides compute capacity for your containers. It also automatically scales for you, resources based on traffic demands.
Requirements/Prerequisites
- An AWS Account.
- Created a User on the account with Permissions to provision resources on the account and Access to CloudWatch.
- Provisioned an AWS EKS Cluster with a managed Node Group.
- Enabled Container Insights for your EKS Cluster by deploying CloudWatch Agent on your cluster.
Part 1: Create the AWS EKS SNS Topic & Subscriptions
We will create two SNS Topics, The Scale Up and Scale Down Topics and add email subscriptions to the topics. The below CloudFormation Templates can be used for this.
The Scale Down Topic and Subscriptions Template.
AWSTemplateFormatVersion: "2010-09-09"
Description: "Template to create SNS Topic and Subscriptions"
Parameters:
Name:
Type: String
Description: The Topic Name
Default: EKS_CLUSTER_MEMORY_LESS_THAN_50
Resources:
Email1Notification:
Type: AWS::SNS::Subscription
Properties:
Endpoint: [email protected]
Protocol: email
TopicArn: !Ref ScaleDownTopic
Email2Notification:
Type: AWS::SNS::Subscription
Properties:
Endpoint: [email protected]
Protocol: email
TopicArn: !Ref ScaleDownTopic
ScaleDownTopic:
Type: AWS::SNS::Topic
Properties:
Tags:
- Key: Name
Value: !Ref Name
- Key: createdBy
Value: Maureen Barasa
- Key: Project
Value: test-blog
TopicName: !Ref Name
Outputs:
ScaleDown:
Description: The scale down topic
Value: !Ref ScaleDownTopic
Export:
Name: "EKSScaleDown"
Value: !Ref ScaleDownTopicN/B: The user should customize the template. The names of Resources, Email endpoints and Tags should match the user’s specific requirements.
The Scale Up Topic and Subscriptions Template
AWSTemplateFormatVersion: "2010-09-09"
Description: "Template to create SNS Topic and Subscriptions"
Parameters:
Name:
Type: String
Description: The Topic Name
Default: EKS_CLUSTER_MEMORY_MORE_THAN_70
Resources:
Email1Notification:
Type: AWS::SNS::Subscription
Properties:
Endpoint:
Protocol: email
TopicArn: !Ref ScaleUpTopic
Email2Notification:
Type: AWS::SNS::Subscription
Properties:
Endpoint:
Protocol: email
TopicArn: !Ref ScaleUpTopic
ScaleUpTopic:
Type: AWS::SNS::Topic
Properties:
Tags:
- Key: Name
Value: !Ref Name
- Key: createdBy
Value:
- Key: Project
Value:
TopicName: !Ref Name
Outputs:
ScaleUp:
Description: The scale up topic
Value: !Ref ScaleUpTopic
Export:
Name: "EKSScaleUp"
Value: !Ref ScaleUpTopic N/B: The user should customize the template. The names of Resources, Email endpoints and Tags should match the user’s specific requirements.
is your first email address e.g [email protected], [email protected] is the second email address e.g [email protected] is the name of a Project to which the Topic applies
Under Outputs we have allowed our values to be exported so we can use them in the template that will create the CloudWatch Alarms.
Once the SNS Topics and Subscriptions are created, go to the given email endpoints and confirm the subscription. We are now ready to create the Scaling Policies and CloudWatch Alarms.
Part 2: Create Scaling Policies and Attach to your AWS EKS NodeGroup ASG
Next, we will create two policies and attach them to the EKS nodegroup autoscaling group. Use the below Cloud formation Template.
AWSTemplateFormatVersion: "2010-09-09"
Description: "create a scaling policy and apply it to EKS ASG Group"
Resources:
ScaleDownPolicy:
Type: AWS::AutoScaling::ScalingPolicy
Properties:
AdjustmentType: "ChangeInCapacity"
PolicyType: "SimpleScaling"
Cooldown: "300"
AutoScalingGroupName: "eks-ASG-GroupName"
ScalingAdjustment: -2
ScaleUpPolicy:
Type: AWS::AutoScaling::ScalingPolicy
Properties:
AdjustmentType: "ChangeInCapacity"
PolicyType: "SimpleScaling"
Cooldown: "300"
AutoScalingGroupName: "eks-ASG-GroupName"
ScalingAdjustment: 2
Outputs:
ScaleUp:
Description: The scale up policy
Value: !Ref ScaleUpPolicy
Export:
Name: "EKSScaleUpPolicy"
Value: !Ref ScaleUpPolicy
ScaleDown:
Description: The scale down policy
Value: !Ref ScaleDownPolicy
Export:
Name: "EKSScaleDownPolicy"
Value: !Ref ScaleDownPolicyPart 3: Create the AWS EKS CloudWatch Alarms
Finally, we will create the CloudWatch Alarms. Here, we will use values created from the above templates. i.e. The SNS Topics and the Scaling Policies.
On the CloudWatch console, select alarms. Then click create alarms.
Under Select Metric, choose “Container Insights” Namespace then select by Cluster name. Under your EKS Cluster name, choose node_memory_utilization metric and click select metric. Next, specify Metric Conditions. For Period one can choose a period from a minute to 1 day. It is the time taken to analyze the metric or expression for creating individual data points for an alarm.
Next, we define the conditions for the alarm. For the first case, we will create an Alarm, where node memory utilization is greater than 70. Click next.
Then we configure actions to take when the alarm is triggered. For, our case our first action is to send a notification to the respective SNS topic created in part 1 above. For example, when the memory utilization average is greater than 70, it should send a notification to EKS_CLUSTER_MEMORY_MORE_THAN_70 SNS topic.
The 2nd action to take will be to scale up based on the scaling policies we defined in part 2 of this article. Ensure to select the EKS cluster node, scaling group. For the high memory utilization alarm, choose the scale-up policy created and click next.
Finally, you create a Name and Description for the Alarms as below and click next. After carefully evaluating your configurations, you can now click, create alarm.
Equally important is to create the alarm for low memory utilization, follow the same steps as above. This time ensure that for the alarm, under conditions, we should select the relevant SNS topic and scaling policies. For example, use the EKS_CLUSTER_MEMORY_LESS_THAN_50 SNS topic and the scale-down policy for scaling actions.
You have now successfully configured EKS cluster auto-scaling using CloudWatch Alarms and SNS topics.
Important Links:
