Security audit using Cloud Custodian for compliance in AWS

Introduction

In this article, we will be talking about Cloud Custodian, an open source rules engine for fleet management in AWS. The simple YAML DSL allows you to easily define rules to enable a well-managed cloud infrastructure, that’s both secure and cost optimized. Cloud Custodian unifies the dozens of tools and scripts most organizations use for managing their AWS accounts into one open source tool. It’s a stateless rules engine for policy definition and enforcement, with metrics and detailed reporting for AWS.

Organizations can use Custodian to manage their AWS environments by ensuring compliance with security policies, tag policies, garbage collection of unused resources, and cost management via off-hours resource management, all from the same place. Custodian policies are written in simple YAML configuration files that specify given resource types and are constructed from a vocabulary of filters and actions.

Installation

Cloud Custodian requires AWS CLI installed and configured to run properly. It is also recommended to install virtualenv before proceeding with the installation. Cloud Custodian can be installed using the following steps, We can clone the contents from the Cloud custodian Github

git clone https://github.com/capitalone/cloud-custodian.git
cd cloud-custodian
pip install -r requirements.txt
python setup.py install

or we can also install it directly using pip

virtualenv --python=python2 custodian
source custodian/bin/activate
(custodian) $ pip install c7n

Now, after installing the Cloud Custodian, we might have to create a new IAM user account with administrative privileges for using it with Custodian. It is recommended to run Custodian within a virtual environment using virtualenv.

Functions of the Cloud Custodian

Cloud Custodian can be used to perform various operations to manage the AWS cloud resources. These can be anything like assigning security settings to resources, to monitoring those resources constantly for changes. Custodian has

• 109 resources
• 524 unique actions
• 376 unique filters

There are also other sections that can be used inside a policy such as description, region, tags, etc.

custodian schema "resource name or blank"

Creating your first policy using Custodian

We will now create a custom custodian policy to turn on instances with the tag ‘Custodian’ to demonstrate the connectivity.

For this demonstration, we will use the instance that we created in our previous AWS IR post. We will use this custodian policy to perform the function,

policies:
  - name: my-first-policy
    resource: ec2
    filters:
      - "tag:Custodian": present
    actions:
      - stop

Let us examine this policy and its functions,

name:This policy will be named as ‘my-first-policy’

resource:This policy will be performing operations on EC2

filters: This policy will filter the resources by checking for a tag named ‘Custodian’ to be present on the system.

action:The resources that were selected after applying the filters will be stopped.

The policy basically stops any EC2 instance running on AWS with a tag ‘Custodian’. We will save this policy as ‘custodian.yml’ inside the Custodian directory.

Validation of the policies in Custodian

Now that we have written our first policy, We can validate the contents of the policy by using the following command

custodian validate custodian.yml

If the syntax of the policy is correct, we will receive an output with ‘Valid’ message as follows,

There is also a dry run option to evaluate the resources from the policy without executing the actions specified in the action: step.

custodian run --dryrun --output-dir . custodian.yml

Executing a policy in Custodian

Let us see how to run this policy in Custodian. Custodian requires an IAM user account with full EC2 and S3 access to run this policy. We will specify the AWS credentials along with the one line command as follows,

AWS_ACCESS_KEY_ID="Key here" AWS_SECRET_ACCESS_KEY="Secret here" custodian run --output-dir=. custodian.yml

We would also have to specify the region of the instance if necessary with the option--region. After running the command, the custodian outputs the following data on the screen.

Custodian stores the logs to the destination specified after the,--output-dir

This simple policy was able to stop a group of instances which were tagged with a ‘Custodian’ tag. The purpose of this policy was to demonstrate the process of writing and executing a policy in Custodian. We will further explore additional functions of the Cloud Custodian with a few more policies.

Demonstration of few other Custodian policies

We will now use additional policies to demonstrate the other possible operations that can be done using Cloud Custodian.

We can download some Custom Cloud Custodian policies from the following GIT repo

git clone https://github.com/tkonduri/CloudCustodianPolicies.git

Now we have a set of general policies that can be used to perform some generic operations.

Automatic tagging of EC2 instances using Lambda functions

Let’s use the following autoTag.yml policy to perform this function. Reviewing the policy,

policies:
  - name: tagging-owner
    resource: ec2
    mode:
        role: arn:aws:iam::521455717944:role/custodian
        type: ec2-instance-state
        events:
          - pending
          - running
          - shutting-down
          - stopping
    filters:
      - "tag:owner": absent 
    actions:
      - type: tag
        key: owner
        value: "surf team.. tagging feature works"

Let’s run the policy in the custodian,

The command executed successfully. In AWS console, we are able to find a Lambda function created to be executed if an instance state change is identified.

As we start an instance, we are able to see that the instance has been tagged automatically.

Auto-tagging instances with the owner id

Let’s use the following owner-autotagging.yml policy to perform this function. Reviewing the policy,

policies:
  - name: owner-autotagging-on_instance_creation
    resource: ec2
    mode:
        role: arn:aws:iam::521455717944:role/custodian
        type: cloudtrail
        events:
          - RunInstances
    filters:
      - tag:owner: absent
    actions:
      - type: auto-tag-user
        tag: owner
        principal_id_tag: CreatorId

Let’s run the policy in the custodian, Below is the screenshot of the output that we receive,

Let’s check AWS lambda for the function which has been created by the custodian

This function will add an owner tag with the instance creator id when the instance is run initiated by cloud trail.

Stop instances with Public IP

This policy in the custodian, publicIP.yml can be used to stop all instances with a public IP address and add a tag to them.

policies:
  - name: custodian_public_ip_check
    resource: ec2
    mode:
        role: arn:aws:iam::521455717944:role/custodian
        type: ec2-instance-state
        events:
          - pending
          - running
          - stopping
    filters:
      - type: value
        key: PublicIpAddress
        op: ne
        value: null
    actions:
      - stop
      - type: tag
        key: custodian_public_ip_check
        value: “surf-custodian....Instance stopped due to PublicIpAddress not allowed”

Now after running the policy in custodian, Below is the screenshot of the output that we receive,

The custodian policy creates a lambda function which will be triggered and the instances will be stopped by the Lambda function.

Conclusion

Thus, Cloud Custodian can be used to manage the AWS resources by setting up automatic functions and triggers to execute operations based on resource events.

Thank you for reading! – Setu Parimi, Steve George

Sign up for the blog directly here.

Check out our professional services here.

Feedback is welcome! For professional services, fan mail, hate mail, or whatever else, contact [email protected]