View amount of instances in AWS CloudWatch / Properly configuring instace scaling

Question

I'm currently developing for a high-traffic website which is using AWS for (among other things) load balancing. We're in the process of configuring the load balancers so we have enough instances running to keep the site afloat, but not too many causing us to lose money.

The way we're approaching this now is by keeping an eye on the CPU utilisation in CloudWatch and changing our configuration accordingly.

A problem we're running into though is that we can't seem to figure out how to get CloudWatch to graph the amount of running instances so we can offset this to the CPU utilisation. We can only see how many instances are currently active so we end up having to check a lot of times instead of just looking back on the statistics of the past hours / days / weeks.

Ideally I'd like to see a graph of amount of instances per Elastic Beanstalk application.

I'm fairly new to AWS and all its components so it could be I'm looking at this in an entirely wrong way, so if anyone has any other suggestions and/or resources on how to properly configure the instance scaling I'd love to hear about it.

Answer 1

I wasn't able to figure out how to see the amount of running instances in CloudWatch. So I decided to measure them myself through the AWS PHP SDK which actually turned out to be much easier than I thought.

Here is the code from my cron command for anyone that wants to know:

public function fire(){
    $client = AWS::get('CloudWatch');

    $containers = [
       'production',
       'staging'
    ];

    foreach($containers as $container){
        $client->putMetricData([
            'Namespace'  => 'Company',
            'MetricData' => [
                [
                    'MetricName' => 'NumberOfInstances',
                    'Timestamp'  => time(),
                    'Value'      => $this->_getInstanceCount($container),
                    'Dimensions' => [
                        [
                            'Name'  => 'Environment',
                            'Value' => AwsHelper::getEnvironment($container)
                        ],
                        [
                            'Name'  => 'Container',
                            'Value' => $container   
                        ]
                    ]
                ]
            ]
        ]);
    }
}

protected function _getInstanceCount($container){
    $beanstalk = AWS::get('ElasticBeanstalk');
    $resources = $beanstalk->describeEnvironmentResources([
        'EnvironmentName' => AwsHelper::getEnvironment($container)
    ])->get('EnvironmentResources');

    return count($resources['Instances']);
}

AwsHelper is a custom class that is only used here to retrieve the environment of a container. AWS is the official AWS SDK for PHP: http://aws.amazon.com/sdkforphp/

I used these docs:

http://docs.aws.amazon.com/aws-sdk-php/latest/class-Aws.CloudWatch.CloudWatchClient.html#_putMetricData

Answer 2

In order to do this Enable Group Metrics Collection on the Auto Scaling Group. To do this go to EC2 Dashboard -> Auto Scaling Groups. Find the group you want to collect metrics on and then use the Enable Group Metrics Collection under the monitoring tab. Once that was completed, you can see the group in CloudWatch under the Scaling Group bucket.

Hope this helps.

Answer 3

Have a read of this post

http://www.techrepublic.com/blog/the-enterprise-cloud/how-to-add-cloudwatch-monitors-to-auto-scale-your-amazon-web-service/#.

Basically you set up the cloudwatch alerts or graphs on your autoscaling group not your ec2 instances. Set it up for cpu average and scale based on the alarms.

Answer 4

To do a similar thing with default EC2/ASG configuration, use the HealthyHostCount metric for the load balancer with which your elastic beanstalk environment is associated.

Assuming all your hosts are healthy, then it'll effectively be the same as the auto scaling group's instance count.

In "CloudWatch > Metrics", search for "healthyhostcount" and you'll be presented with a list of load balancers. Select the one for the EB environment you want to monitor.