Auto scaling using cloud formation according to Request count
문제
We are using cloud formation for auto scaling according based upon the load balancer RequestCount metric. Currently we scale up an instance if the request is increased to 1500 for 1 minute (each instance can handle 1500 request per minute). The problem is, since the autoscaling group continually checks the RequestCount and adds a new instance if the request count is greater than 1500 for 1 min. But it is not required as I now have 2 instances which can handle 3000 req. per minute. Is there any facility to make matrices custom? i.e. if new instance is added then the scale up policy will change to 3000 req.
Example scenario:
- Initially there is 1 ELB, 1 tomcat instance attached to ELB(can handle 1500 req. per min).
- 1 cloud watch with action of scale up ploicy if the req. count on ELB is increased to 1500 for min.
- Currently request load on ELB is 1500 for 1 min. now req. load is increased to 1700 for min. so it will attach a new tomcat instance on ELB. So i have 2 instance which can handle 3000 req. for min.
- But now what problem is cloud watch still check the req. count on ELB and if req. load is 1700 for min. it will add one new tomcat instance which is not required.
How can i over come from this problem?
해결책
What you want to do is use the average for the load balancer. You can have different types of metrics. Sum, Average, Minimum, Maximum and Sample. If you choose Average it will give you an average for all the instances under the loadbalancer. So it will only trigger a new instance launch when all the servers in your group are at 1500 requests per minute.
Quick description of the type:
- Average - Average for the load balancer
- Sum - The total number of request (example: 3000)
- Maximum - The max number of requests any server has (because it might not be balanced exactly)
- Minimum - The min number of requests any server has (because it might not be balanced exactly)
- Sample - The number of servers used to calculate the average (essentially how many servers are on the load balancer)
You can create you're own custom metrics as well but you'll need to create an application that tells amazon what the values are. Using the cloud watch api you can easily create your own. Take a look here http://docs.amazonwebservices.com/AmazonCloudWatch/latest/DeveloperGuide/Welcome.html?r=1540
다른 팁
As dsummersl said, RequestCount cannot be used straightaway with Average metrics.
I found a couple of workarounds here. Basically, they tell you to define a custom CloudWatch metric so you can instrument your instances to emit the number of requests that they are receiving.
For this you will need to gather the request count from each instance and send it to cloudwatch using put-metric-data, or creating an ebextension so the EBS do it for you (I think this is the simplest way).
Then you can set Auto Scaling to scale up/down based on that custom metric.
Hope it helps.
I've made the scaling based on RequestCount
work by applying Scaling Policy with associated Cloudwatch alarms that trigger policy evaulation. Below you'll find cloudformation template that I've used in ElasticBeanstalk app:
RequestCountScalingAlarmLt2000:
Type: "AWS::CloudWatch::Alarm"
Properties:
ActionsEnabled: true
AlarmActions:
- Ref: RequestCountScalingPolicy
OKActions:
- Ref: RequestCountScalingPolicy
AlarmDescription: "Scale when Request Count < 2000"
AlarmName: {"Fn::Join": ["-", ["Scale when Request Count < 2000", { "Ref":"AWSEBEnvironmentName" }]]}
ComparisonOperator: LessThanThreshold
Dimensions:
- Name: LoadBalancerName
Value:
Ref: AWSEBLoadBalancer
EvaluationPeriods: "1"
MetricName: RequestCount
Namespace: AWS/ELB
Period: "300"
Statistic: Sum
Threshold: 2000
RequestCountScalingAlarmGt2000:
Type: "AWS::CloudWatch::Alarm"
Properties:
ActionsEnabled: true
AlarmActions:
- Ref: RequestCountScalingPolicy
OKActions:
- Ref: RequestCountScalingPolicy
AlarmDescription: "Scale when 2000 < Request Count < 20000"
AlarmName: "Scale when Request 2000 < Count < 20000"
ComparisonOperator: GreaterThanOrEqualToThreshold
Dimensions:
- Name: LoadBalancerName
Value:
Ref: AWSEBLoadBalancer
EvaluationPeriods: "1"
MetricName: RequestCount
Namespace: AWS/ELB
Period: "300"
Statistic: Sum
Threshold: 2000
RequestCountScalingAlarmGt20000:
Type: "AWS::CloudWatch::Alarm"
Properties:
ActionsEnabled: true
AlarmActions:
- Ref: RequestCountScalingPolicy
OKActions:
- Ref: RequestCountScalingPolicy
AlarmDescription: "Scale when 20000 < Request Count < 30000"
AlarmName: "Scale when 20000 < Request Count < 30000"
ComparisonOperator: GreaterThanOrEqualToThreshold
Dimensions:
- Name: LoadBalancerName
Value:
Ref: AWSEBLoadBalancer
EvaluationPeriods: "1"
MetricName: RequestCount
Namespace: AWS/ELB
Period: "300"
Statistic: Sum
Threshold: 20000
RequestCountScalingAlarmGt30000:
Type: "AWS::CloudWatch::Alarm"
Properties:
ActionsEnabled: true
AlarmActions:
- Ref: RequestCountScalingPolicy
OKActions:
- Ref: RequestCountScalingPolicy
AlarmDescription: "Scale when 30000 < Request Count < 40000"
AlarmName: "Scale when 30000 < Request Count < 40000"
ComparisonOperator: GreaterThanOrEqualToThreshold
Dimensions:
- Name: LoadBalancerName
Value:
Ref: AWSEBLoadBalancer
EvaluationPeriods: "1"
MetricName: RequestCount
Namespace: AWS/ELB
Period: "300"
Statistic: Sum
Threshold: 30000
RequestCountScalingAlarmGt40000:
Type: "AWS::CloudWatch::Alarm"
Properties:
ActionsEnabled: true
AlarmActions::
- Ref: RequestCountScalingPolicy
OKActions:
- Ref: RequestCountScalingPolicy
AlarmDescription: "Scale when 40000 < Request Count < 50000"
AlarmName: "Scale when 40000 < Request Count < 50000"
ComparisonOperator: GreaterThanOrEqualToThreshold
Dimensions:
- Name: LoadBalancerName
Value:
Ref: AWSEBLoadBalancer
EvaluationPeriods: "1"
MetricName: RequestCount
Namespace: AWS/ELB
Period: "300"
Statistic: Sum
Threshold: 40000
RequestCountScalingAlarmGt50000:
Type: "AWS::CloudWatch::Alarm"
Properties:
ActionsEnabled: true
AlarmActions:
- Ref: RequestCountScalingPolicy
OKActions:
- Ref: RequestCountScalingPolicy
AlarmDescription: "Scale when 50000 < Request Count < 60000"
AlarmName: "Scale when 50000 < Request Count < 60000"
ComparisonOperator: GreaterThanOrEqualToThreshold
Dimensions:
- Name: LoadBalancerName
Value:
Ref: AWSEBLoadBalancer
EvaluationPeriods: "1"
MetricName: RequestCount
Namespace: AWS/ELB
Period: "300"
Statistic: Sum
Threshold: 50000
RequestCountScalingAlarmGt60000:
Type: "AWS::CloudWatch::Alarm"
Properties:
ActionsEnabled: true
AlarmActions::
- Ref: RequestCountScalingPolicy
OKActions:
- Ref: RequestCountScalingPolicy
AlarmDescription: "Scale when 60000 < Request Count < 70000"
AlarmName: "Scale when 60000 < Request Count < 70000"
ComparisonOperator: GreaterThanOrEqualToThreshold
Dimensions:
- Name: LoadBalancerName
Value:
Ref: AWSEBLoadBalancer
EvaluationPeriods: "1"
MetricName: RequestCount
Namespace: AWS/ELB
Period: "300"
Statistic: Sum
Threshold: 60000
RequestCountScalingAlarmGt70000:
Type: "AWS::CloudWatch::Alarm"
Properties:
ActionsEnabled: true
AlarmActions:
- Ref: RequestCountScalingPolicy
OKActions:
- Ref: RequestCountScalingPolicy
AlarmDescription: "Scale when Request Count >= 70000"
AlarmName: "Scale when Request Count >= 70000"
ComparisonOperator: GreaterThanOrEqualToThreshold
Dimensions:
- Name: LoadBalancerName
Value:
Ref: AWSEBLoadBalancer
EvaluationPeriods: "1"
MetricName: RequestCount
Namespace: AWS/ELB
Period: "300"
Statistic: Sum
Threshold: 70000
RequestCountScalingPolicy:
Type: "AWS::AutoScaling::ScalingPolicy"
Properties:
AutoScalingGroupName:
Ref: "AWSEBAutoScalingGroup"
AdjustmentType: "ExactCapacity"
PolicyType: "StepScaling"
EstimatedInstanceWarmup: 120
StepAdjustments:
-
MetricIntervalLowerBound: "0"
MetricIntervalUpperBound: "2000"
ScalingAdjustment: "1"
-
MetricIntervalLowerBound: "2000"
MetricIntervalUpperBound: "20000"
ScalingAdjustment: "2"
-
MetricIntervalLowerBound: "20000"
MetricIntervalUpperBound: "30000"
ScalingAdjustment: "3"
-
MetricIntervalLowerBound: "30000"
MetricIntervalUpperBound: "40000"
ScalingAdjustment: "4"
-
MetricIntervalLowerBound: "40000"
MetricIntervalUpperBound: "50000"
ScalingAdjustment: "5"
-
MetricIntervalLowerBound: "50000"
MetricIntervalUpperBound: "60000"
ScalingAdjustment: "6"
-
MetricIntervalLowerBound: "60000"
MetricIntervalUpperBound: "70000"
ScalingAdjustment: "7"
-
MetricIntervalLowerBound: "70000"
ScalingAdjustment: "8"