Bringing the Ansible development to the cloud

How to use Vagrant to smoothly bring the Ansible development environment to the cloud

It’s very important that you, as a sysadmin, have your own environment, where you can develop and test Ansible playbooks. Like any dev guy’s environment, your environment must be of your total control, because you will certainly need to recreate it from the scratch many times. The environment must be not shared as well, therefore with no risk of being on an unexpected state, after someone intervention.

Vagrant is an excellent tool to manage the Ansible development environment. Its default integration with VirtualBox, amongst other hypervisors, allows you to have virtual machines in your own host. Through its simple command-line interface, you are able to create and destroy them, over and over, at your will.

Vagrant uses specific images, known as boxes. Most of them you can find in Vagrant Cloud. There are Vagrant boxes for several providers, like VirtualBox. In addition, there are boxes of all sort of Linux distributions, as well as with other open source software installed. You too can provision your local virtual machine with software and configuration, package it as a Vagrant box and share it in Vagrant Cloud, as explained in the article Choosing between baked and fried provisioning.

Besides handling your local virtual machines, Vagrant can also manage EC2 instances in AWS. If you have hardware limitations, why not benefit from the cloud? If your environment demands more resources than the available on your host, it’s a great idea to bring the Ansible development environment to AWS. Don’t forget that AWS charge you, if you are not eligible to the AWS Free Tier or if you have exceeded your free tier usage limit.

AWS Setup

In order to bring the Ansible development environment to AWS, you must follow some steps:

  1. First of all, you must have an AWS account;
  2. Then, you must create an user with full access to the EC2 resources (eg: AmazonEC2FullAccess permission) through the IAM (Identity and Access Management) console;
  3. After that, you must create an access key in the Security credential tab. Set the local environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with the respective values from the access key id and the secret access key, generated during the access key creation;
  4. Alternatively, install the AWS CLI (Command Line Interface) tool and configure your local environment through the command aws configure. This command prompts for information about the access key and the default profile’s region, storing them inside the .aws directory of your home folder.

AWS Region Setup

On AWS, you can create EC2 instances in different regions. You must choose one specific region to create your EC2 instances at a time. It can be done by defining the AWS_REGION or the EC2_REGION environment variables, or through the cited aws configure command. The environment variables have precedence, when both configuration are made.

Before creating the EC2 instances, you must create a security group and key pairs in the chosen region. You can do it:

  • manually, through the EC2 console;
  • through the command line, by using the AWS CLI tool;
  • programmatically, by using the Amazon EC2 API;
  • automatically, by using Ansible AWS modules!

The playbook-aws-region-configuration.yml file below is a good example of using Ansible to automate the configuration of a specific AWS region. The playbook is responsible for creating the required resources and gathering information from the AWS region for later use by Vagrant. If you want to run the Codeyourinfra project’s solutions on AWS, you must execute the playbook previously, for your chosen AWS region.

---
- hosts: localhost
  connection: local
  gather_facts: false
  vars_prompt:
    - name: "aws_region"
      prompt: "AWS Region"
      default: "sa-east-1"
      private: no
  tasks:
  - name: Create the AWS directory if it doesn't exist
    file:
      path: '{{aws_region}}'
      state: directory
  - name: Get the VPCs
    ec2_vpc_net_facts:
      region: '{{aws_region}}'
    register: ec2_vpc_net_facts_results
  - name: Create the Vagrant security group
    ec2_group:
      name: vagrant
      description: Security Group for EC2 instances managed by Vagrant
      region: '{{aws_region}}'
      vpc_id: '{{default_vpc.id}}'
      rules:
        - proto: tcp
          ports:
            - 22
            - 80
            - 3000
            - 8080
            - 8086
          cidr_ip: 0.0.0.0/0
    vars:
      default_vpc: '{{(ec2_vpc_net_facts_results|json_query("vpcs[?is_default]"))[0]}}'
    register: ec2_group_result
  - name: Store the security group's data
    copy:
      content: '{{ec2_group_result|to_nice_json}}'
      dest: '{{aws_region}}/security-group.json'
  - name: Get the default VPC's subnets
    ec2_vpc_subnet_facts:
      region: '{{aws_region}}'
      filters:
        vpc-id: '{{ec2_group_result.vpc_id}}'
    register: ec2_vpc_subnet_facts_results
  - name: Store the VPC subnet's data
    copy:
      content: '{{(ec2_vpc_subnet_facts_results.subnets|sort(attribute="availability_zone"))[0]|to_nice_json}}'
      dest: '{{aws_region}}/subnet.json'
  - name: Create the key pairs
    ec2_key:
      name: codeyourinfra-aws-key
      region: '{{aws_region}}'
    register: ec2_key_result
  - name: Store the private key
    copy:
      content: '{{ec2_key_result.key.private_key}}'
      dest: '{{aws_region}}/codeyourinfra-aws-key.pem'
      mode: 0400
    when: ec2_key_result.key.private_key is defined
  - name: Find Ubuntu Server 14.04 LTS AMIs
    ec2_ami_find:
      name: 'ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-*'
      region: '{{aws_region}}'
      owner: 099720109477
      sort: name
      sort_order: descending
      sort_end: 1
    register: ec2_ami_find_result
  - name: Store the Ubuntu AMI's data
    copy:
      content: '{{ec2_ami_find_result.results[0]|to_nice_json}}'
      dest: '{{aws_region}}/ubuntu-ami.json'
    when: ec2_ami_find_result.results[0] is defined

Behind the scenes the used Ansible modules interact with the Amazon EC2 API. Here are some details about the playbook:

  • the ec2_vpc_net_facts module is used to get the default VPC (Virtual Private Cloud) of the chosen region;
  • the ec2_group module is used to create the required security group, whose data is stored in the security-group.json file, for later use by Vagrant;
  • the ec2_vpc_subnet_facts module is used to select a subnet, and store its data in the subnet.json file, for later use by Vagrant;
  • the ec2_key module is used to create the required key pairs, and store the private key in the codeyourinfra-aws-key.pem file, for later use by Vagrant;
  • the ec2_ami_find module is used to select an Ubuntu AMI (Amazon Machine Image), and store its data in the ubuntu-ami.json file, for later use by Vagrant.

If you haven’t cloned yet the Codeyourinfra project’s repository, do it right now 🙂 You will find the playbook-aws-region-configuration.yml file inside the cloud/aws directory. Go to the folder and run the following command, informing your AWS region of preference when prompted:

ansible-playbook playbook-aws-region-configuration.yml

Vagrant up

In order to make Vagrant manage EC2 instances, you must install the AWS plugin. Execute the command vagrant plugin install vagrant-aws. You can find details about the plugin on its Github repository.

Every Codeyourinfra project’s solution has an aws subdirectory, where is placed the specific Vagrantfile for managing EC2 instances.  One example is the Vagrantfile below, that creates on AWS the Ansible development environment of the solution explained in the article How to unarchive different files in different servers in just one shot.

Notice that the Vagrantfile handles the environment variables introduced in the Codeyourinfra project’s release 1.4. The APPEND_TIMESTAMP and the PROVISIONING_OPTION environment variables are explained in detail by the blog post Choosing between fried and baked provisioning.

If you initialize the EC2 environment with the baked provisioning option, the São Paulo region (sa-east-1) will be used, because it is the region where the repo server AMI (ami-b86627d4) is available. Otherwise, the AWS region where the EC2 instances will be created is taken from either the AWS_REGION or the EC2_REGION environment variable.

# -*- mode: ruby -*-
# vi: set ft=ruby :

load File.join("..", "..", "common", "timestamp-appender.rb")

provisioning_option = ENV['PROVISIONING_OPTION'] || "fried"
if provisioning_option != "baked" && provisioning_option != "fried"
  puts 'PROVISIONING_OPTION must be \'baked\' or \'fried\'.'
  abort
end

if provisioning_option == "baked"
  aws_region = "sa-east-1"
else
  aws_region = ENV['AWS_REGION'] || ENV['EC2_REGION'] || "sa-east-1"
end
relative_path = File.join("..", "..", "cloud", "aws", aws_region)
security_group  = JSON.parse(File.read(File.join(relative_path, "security-group.json")))
subnet = JSON.parse(File.read(File.join(relative_path, "subnet.json")))
ubuntu_ami = JSON.parse(File.read(File.join(relative_path, "ubuntu-ami.json")))
ec2_instances = JSON.parse('[{"name": "repo", "role": "repository"}, {"name": "server1", "role": "server"}, {"name": "server2", "role": "server"}]')

Vagrant.configure("2") do |config|
  config.vm.box = "dummy"
  config.vm.box_url = "https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box"
  config.vm.synced_folder ".", "/vagrant", disabled: true

  ec2_instances.each do |ec2_instance|
    config.vm.define ec2_instance["name"] do |ec2_config|
      ec2_config.vm.provider "aws" do |aws, override|
        aws.region = aws_region
        if ec2_instance["name"] == "repo" && provisioning_option == "baked"
          aws.ami = "ami-b86627d4"
        else
          aws.ami = ubuntu_ami['ami_id']
        end
        aws.instance_type = "t2.micro"
        aws.keypair_name = "codeyourinfra-aws-key"
        aws.security_groups = security_group['group_id']
        aws.subnet_id = subnet['id']
        aws.tags = {"Name" => ec2_instance["name"], "Role" => ec2_instance["role"], "Solution" => "unarchive_from_url_param"}
        override.ssh.username = "ubuntu"
        override.ssh.private_key_path = File.join(relative_path, "codeyourinfra-aws-key.pem")
        override.nfs.functional = false
      end
      if ec2_instance["name"] == "repo" && provisioning_option == "fried"
        ec2_config.vm.provision "ansible" do |ansible|
          ansible.playbook = File.join("..", "playbook-repo.yml")
        end
      end
    end
  end
end

Besides the region, other required AWS provider-specific configuration options are defined:

  • ami – the AMI id to boot. If you initialize the environment with the baked provisioning option, the AMI is the one previously prepared, as mentioned.  (If you have installed the AWS CLI tool and would like to know the AMIs provided by the Codeyourinfra project, just execute the command aws ec2 describe-images –owners 334305766942Otherwise, the AMI is the one selected during the AWS Region Setup phase, obtained from the ubuntu-ami.json file.
  • instance_type – AWS provides a wide range of EC2 instance types, for different use cases. For our testing purposes, the T2 instances are more than sufficient. Besides that, the t2.micro instance type is eligible to the AWS Free Tier.
  • keypair_name – the name of keypair created during the AWS Region Setup phase, when the playbook-aws-region-configuration.yml was executed. The path of the stored private key file (codeyourinfra-aws-key.pem) is then configured by overriding the default Vagrant ssh.private_key_path configuration.
  • security_groups – the id of the security group created during the AWS Region Setup phase, obtained from the security-group.json file.  The security group was created exclusively for EC2 instances managed by Vagrant.
  • subnet_id – the id of the subnet selected during the AWS Region Setup phase, obtained from the subnet.json file. The subnet was selected from the default VPC’s subnets list, ordered by availability zone.
  • tags – a hash of tags to set on the EC2 instance. The tags are very useful for later EC2 instances identification.

Now that you have checked out the Vagrantfile which is in the unarchive_from_url_param/aws directory of the Codeyourinfra project’s repository, stay there and run the command below in order to see the magic happens!

vagrant up

Ansible inventory

The Ansible inventory file is where you group your machines. Ansible needs the information placed there in order to connect to the hosts through SSH. It enables the agentless characteristic of the tool and makes possible a task be executed in several servers in a single execution.

When you use AWS, each time you create an EC2 instance, it gets a different IP address. Differently than when you create local virtual machines, you are not able to define the IP address in the Vagrantfile. The Vagrant AWS plugin highlights in the output: “Warning! The AWS provider doesn’t support any of the Vagrant high-level network configurations (‘config.vm.network‘). They will be silent ignored.”

For that reason, inside the aws directory below every Codeyourinfra project folder in the repository, you will find two more files: playbook-ec2-instances-inventory.yml and ec2_hosts_template. The Ansible playbook is responsible for discovering the IP addresses of the just created EC2 instances and, based on the template, generating the ec2_hosts file.

You must execute the playbook right after the EC2 instances have been created. Just run:

ansible-playbook playbook-ec2-instances-inventory.yml

Once generated the ec2_hosts file, you can use it as the inventory option (-i) of either ansible or ansible-playbook commands. For example, run the following Ansible ad-hoc command:

ansible ec2_instances -m ping -i ec2_hosts

It will use the ping module to test the connection with all of the EC2 instances up and running.

Ansible development

Finally, you have your Ansible development environment on AWS. You can edit any playbook and test them against the created EC2 instances. If something goes wrong, you can recreate the entire environment from the scratch. You have the autonomy to do what you want, because the environment is yours.

The next step is automate the test of your infrastructure code. But it’s the subject of the next article. Stay tuned!

Before I forget, I must reinforce it: the purpose of the Codeyourinfra project is help you. So, don’t hesitate to tell the problems you face as a sysadmin.

3 Replies to “Bringing the Ansible development to the cloud”

  1. Hello ,

    I saw your tweets and thought I will check your website. Have to say it looks very good!
    I’m also interested in this topic and have recently started my journey as young entrepreneur.

    I’m also looking for the ways on how to promote my website. I have tried AdSense and Facebok Ads, however it is getting very expensive.
    Can you recommend something what works best for you?

    I also want to improve SEO of my website. Would appreciate, if you can have a quick look at my website and give me an advice what I should improve: http://janzac.com/
    (Recently I have added a new page about FutureNet and the way how users can make money on this social networking portal.)

    I have subscribed to your newsletter. 🙂

    Hope to hear from you soon.

    P.S.
    Maybe I will add link to your website on my website and you will add link to my website on your website? It will improve SEO of our websites, right? What do you think?

    Regards
    Jan Zac

    1. Hi Jan Zac,

      I recommend the Yoast SEO plugin. Check it out!

      I’ve taken a look at your website and I really enjoyed. My advice is: be consistent on creating great content to solve your audience problems!

      Regards,

      Gustavo.

Leave a Reply

Your email address will not be published. Required fields are marked *