In this article, we will be talking about Cloud Custodian, an open source rules engine for fleet management in AWS. The simple YAML DSL allows you to easily define rules to enable a well-managed cloud infrastructure, that’s both secure and cost optimized. Cloud Custodian unifies the dozens of tools and scripts most organizations use for managing their AWS accounts into one open source tool. It’s a stateless rules engine for policy definition and enforcement, with metrics and detailed reporting for AWS.
Organizations can use Custodian to manage their AWS environments by ensuring compliance with security policies, tag policies, garbage collection of unused resources, and cost management via off-hours resource management, all from the same place. Custodian policies are written in simple YAML configuration files that specify given resource types and are constructed from a vocabulary of filters and actions.
Cloud Custodian requires AWS CLI installed and configured to run properly. It is also recommended to install virtualenv before proceeding with the installation. Cloud Custodian can be installed using the following steps, We can clone the contents from the Cloud custodian Github
git clone https://github.com/capitalone/cloud-custodian.git cd cloud-custodian pip install -r requirements.txt python setup.py install
or we can also install it directly using pip
virtualenv --python=python2 custodian source custodian/bin/activate (custodian) $ pip install c7n
Now, after installing the Cloud Custodian, we might have to create a new IAM user account with administrative privileges for using it with Custodian. It is recommended to run Custodian within a virtual environment using virtualenv.
Functions of the Cloud Custodian
Cloud Custodian can be used to perform various operations to manage the AWS cloud resources. These can be anything like assigning security settings to resources, to monitoring those resources constantly for changes. Custodian has
- 109 resources
- 524 unique actions
- 376 unique filters
name:– Used to specify the name of the policy
resource:– Used to specify the resource to be used with the policy out of the 109 available resources.
mode:– Used to specify the mode of operation such as Poll (Default), Cloud Trail, Periodic, Config Rule
filters:– Used to specify filters to select the expected resource.
actions:– Action to be performed on the resource out of the 524 available actions.
There are also other sections that can be used inside a policy such as description, region, tags, etc.
custodian schema "resource name or blank"
Creating your first policy using Custodian
We will now create a custom custodian policy to turn on instances with the tag ‘Custodian’ to demonstrate the connectivity.
For this demonstration, we will use the instance that we created in our previous AWS IR post. We will use this custodian policy to perform the function,
policies: - name: my-first-policy resource: ec2 filters: - "tag:Custodian": present actions: - stop
Let us examine this policy and its functions,
name:This policy will be named as ‘my-first-policy’
resource:This policy will be performing operations on EC2
filters: This policy will filter the resources by checking for a tag named ‘Custodian’ to be present on the system.
action:The resources that were selected after applying the filters will be stopped.
The policy basically stops any EC2 instance running on AWS with a tag ‘Custodian’. We will save this policy as ‘custodian.yml’ inside the Custodian directory.
Validation of the policies in Custodian
Now that we have written our first policy, We can validate the contents of the policy by using the following command
custodian validate custodian.yml
If the syntax of the policy is correct, we will receive an output with ‘Valid’ message as follows,
There is also a dry run option to evaluate the resources from the policy without executing the actions specified in the
custodian run --dryrun --output-dir . custodian.yml
Executing a policy in Custodian
Let us see how to run this policy in Custodian. Custodian requires an IAM user account with full EC2 and S3 access to run this policy. We will specify the AWS credentials along with the one line command as follows,
AWS_ACCESS_KEY_ID="Key here" AWS_SECRET_ACCESS_KEY="Secret here" custodian run --output-dir=. custodian.yml
We would also have to specify the region of the instance if necessary with the option
--region. After running the command, the custodian outputs the following data on the screen.
Custodian stores the logs to the destination specified after the,
This simple policy was able to stop a group of instances which were tagged with a ‘Custodian’ tag. The purpose of this policy was to demonstrate the process of writing and executing a policy in Custodian. We will further explore additional functions of the Cloud Custodian with a few more policies.
Demonstration of few other Custodian policies
We will now use additional policies to demonstrate the other possible operations that can be done using Cloud Custodian.
We can download some Custom Cloud Custodian policies from the following GIT repo
git clone https://github.com/tkonduri/CloudCustodianPolicies.git
Now we have a set of general policies that can be used to perform some generic operations.
Automatic tagging of EC2 instances using Lambda functions
Let’s use the following autoTag.yml policy to perform this function. Reviewing the policy,
policies: - name: tagging-owner resource: ec2 mode: role: arn:aws:iam::521455717944:role/custodian type: ec2-instance-state events: - pending - running - shutting-down - stopping filters: - "tag:owner": absent actions: - type: tag key: owner value: "surf team.. tagging feature works"
Let’s run the policy in the custodian,
The command executed successfully. In AWS console, we are able to find a Lambda function created to be executed if an instance state change is identified.
As we start an instance, we are able to see that the instance has been tagged automatically.
Auto-tagging instances with the owner id
Let’s use the following owner-autotagging.yml policy to perform this function. Reviewing the policy,
policies: - name: owner-autotagging-on_instance_creation resource: ec2 mode: role: arn:aws:iam::521455717944:role/custodian type: cloudtrail events: - RunInstances filters: - tag:owner: absent actions: - type: auto-tag-user tag: owner principal_id_tag: CreatorId
Let’s run the policy in the custodian, Below is the screenshot of the output that we receive,
Let’s check AWS lambda for the function which has been created by the custodian
This function will add an owner tag with the instance creator id when the instance is run initiated by cloud trail.
Stop instances with Public IP
This policy in the custodian, publicIP.yml can be used to stop all instances with a public IP address and add a tag to them.
policies: - name: custodian_public_ip_check resource: ec2 mode: role: arn:aws:iam::521455717944:role/custodian type: ec2-instance-state events: - pending - running - stopping filters: - type: value key: PublicIpAddress op: ne value: null actions: - stop - type: tag key: custodian_public_ip_check value: “surf-custodian....Instance stopped due to PublicIpAddress not allowed”
Now after running the policy in custodian, Below is the screenshot of the output that we receive,
The custodian policy creates a lambda function which will be triggered and the instances will be stopped by the Lambda function.
Thus, Cloud Custodian can be used to manage the AWS resources by setting up automatic functions and triggers to execute operations based on resource events.
Thank you for reading! – Setu Parimi, Steve George
Sign up for the blog directly here.
Check out our professional services here.
Feedback is welcome! For professional services, fan mail, hate mail, or whatever else, contact [email protected]