How to provision a local VM and remote amazon EC2 instance with the same Chef and Vagrant setup

20th Dec, 2014 | chef devops ec2 vagrant virtualbox

In the previous article we learned how to create a local virtual machine for development and a similar live server on Amazon EC2 with Vagrant.

That helped us setup the servers and get going with Vagrant but we didn't install anything on them. So let's do that now!

First a recap of the tools we are using:

Vagrant - The glue that holds the whole process together. Vagrant co-ordinates virtualisation providers to manage virtual servers and provisioners that will then load apps and tweak settings on those servers.

Virtualbox - The virtualisation provider for local virtual machines. Also the default for vagrant, but other providers can be used.

Vagrant EC2 plugin - The link to the virtualisation provider for servers on the Amazon EC2 platform.

Chef - The tool to add applications, modules and config files to the server that is controlled by Vagrant. The provisioner.

The good thing about this toolset is they all abstract their work domain well. Vagrant can work with different virtualisation providers, such as Virtualbox or VMware. It can use different provisioners such as Chef or Puppet. With whatever combinations you still use the same vagrant instructions to work - vagrant up, vagrant destroy, vagrant provision, vagrant shh.

Chef abstracts the provisioning process so the same Chef configuration can be used for whatever type of server you wish to cook up (sorry!). In theory this is true, but in practise it may need a bit of OS specific config here and there. To be fair this stuff is HARD so sometimes you have to be aware that you have a particular strain of a certain OS. There might be a way around this but in my last setup to install a certain application on Ubuntu I had to ensure apt-get update was called before the app was installed. But I could do this with Chef, so it still keeps the philosophy of the toolset.

And the philosophy of the toolset? To be able to produce a portable and reproducible development environment.

And this is what I want to do. To be able to produce a local development server and then reproduce this server on EC2. In the previous article we created a Vagrant managed server both locally and on EC2. So here we now need to feed these servers some food - in the form of apps and config.

Our shopping list of tasks will be:

  • Install Chef
  • Setup the local chef directory structure
  • Create a git repo for the whole project (the chef tools manage cookbooks via git)
  • Add some cookbooks
  • Instruct vagrant to provision the VM with Chef to install MySQL and PHP
  • Create a custom cookbook to set the root MySQL password, create a user and database and populate from a dump file.
  • Repeat on our remote EC2 server to provision with the same setup as the development machine

 

Installing chef

First step is follow instructions here:

https://docs.chef.io/chef_solo/

The aim is to just install chef on your machine and go no further into configuration. We will be using chef-solo, which means all configuration will be kept and managed locally. That's fine for this project, we can keep our config close. The other types of chef are suited for managing multiple clusters of servers which sounds like an adventure for another day.

To keep things simple we won't even have to call chef-solo commands ourself. Vagrant will do that. The one chef tool we will have to use is called 'knife' which is used to carve up config files, or cookbooks to use the Chef terminology.

Installing cookbooks

Before we start let's recap the file structure of our project so far:

.
├── vagrant_ec2
│   └── Vagrantfile
└── vagrant_local
       └── Vagrantfile

Let's start by asking chef to install php and mysql for us on our VM. To do this we have to use knife to install the cookbooks for php and mysql. We will then instruct vagrant to tell chef to run recipes from those cookbooks.

One thing to be aware of with using knife (hold the blunt end?) is it requires a git repo to work with. But we were going to put our server config into a repo anyway, so let's do it now.

mkdir chef
cd chef
mkdir cookbooks
touch cookbooks/.gitignore
cd ..
git init
git add -A
git commit -m "First commit"

Now we can start using knives:

cd chef
knife cookbook site install apt
knife cookbook site install php
knife cookbook site install mysql
knife cookbook site install iis
knife cookbook site install yum-epel

Ideally we would only be installing the php and mysql cookbooks but we need a few extras to smooth over. After all this stuff is tricky to do across such a wide range of platforms. The apt cookbook will ensure our Ubuntu server is up to date when we start installing, the iis and yum-epel keep the other cookbooks happy.

During the install your screen should show you knife doing lots of stuff. If you look in your cookbook directory you will see the downloaded cookbooks:

.
└── cookbooks
  ├── apt
  ├── build-essential
  ├── chef-sugar
  ├── chef_handler
  ├── iis
  ├── mysql
  ├── php
  ├── windows
  ├── xml
  ├── yum
  ├── yum-epel
  └── yum-mysql-community

Cookbooks are packages that can have dependencies on other cookbooks. knife is clever enough to deal with these dependencies and load them for us, which accounts for the extras here (beyond our own extra's we specified).

Getting Vagrant to read cookbooks

Now we can edit the Vagrantfile for our VM. Refer to the final copy of the file at the bottom of the article to see where to fit things in. Here we tell vagrant to use chef for provisioning and which recipes to run. We also need to tell vagrant where the chef cookbooks are:

# Use chef to provision this machine

  config.vm.provision "chef_solo" do |chef|

    # Cookbooks directory path relative to this file
    chef.cookbooks_path = ["../chef/cookbooks"]

    # Specify recipes we want 'cooked'. Format is cookbook::recipe
    chef.add_recipe "apt::default"
    chef.add_recipe "php::default"
    chef.add_recipe "php::module_mysql"
    chef.add_recipe "mysql::server"
    chef.add_recipe "mysql::client"

  end

Earlier chef was installed on your host machine so cookbooks could be downloaded, but we

also need chef to be installed on the virtual machine too. The chef client on the target machine is sometimes included in base boxes, so may already be there but that is not

guaranteed. Luckily there is a vagrant plugin that will ensure chef is installed on the target machine, and if not install it for us. To install the plugin run in your shell:

vagrant plugin install vagrant-omnibus

And then update your Vagrantfile to use the plugin:

config.omnibus.chef_version = :latest

Provision the local VM

Now from the vagrant_local directory tell vagrant to provision the server.

vagrant reload --provision

Again chef will fill your screen in green with it's activity. Once completed you can

login and verify it's installed mysql

vagrant ssh

mysql --version

> mysql Ver 14.14 Distrib 5.5.40, for debian-linux-gnu (x86_64) using readline 6.3

php -v

PHP 5.5.9-1ubuntu4.5 (cli) (built: Oct 29 2014 11:59:10)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies
with Zend OPcache v7.0.3, Copyright (c) 1999-2014, by Zend Technologies

Result! Chef has cooked up this stack for us.

Passing parameters to cookbook recipes

What we've done so far is run off the shelf cookbooks that install standard packages.

We haven't yet told Chef about anything specific about our particular install. Cookbooks often contain multiple recipes so you can customise an install by selecting appropriate recipes. For example if we only needed MySQL client we would of left out the MySQL server recipe. The other way to customise chefs actions is to pass in cookbook parameters. There's often a wide range of cookbook parameters which you can find detailed in the cookbook docs. Let's start by specifying the root password for mysql (from a security point of view this is not a production solution, just a demo). We can do this by passing the value to the mysql cookbook in our Vagrantfile:

chef.json = {
  "mysql" => {
    "server_root_password" => "secure_password"
  }
}

And ask vagrant to shake this change through to the VM

vagrant provision

This command runs Chef on an already running VM. A core principle of Chef is it's operation is idempotent - running it multiple times will result in the same outcome. In this case the root password gets updated but everything else stays the same. This is great for developing our setup, we can make small steps and test each time.

Creating custom cookbooks

So next something more adventurous. We will setup a database, user and then import data into the database from a dump file so our app has an initial state. To my knowledge this isn't possible with the default cookbook so lets create our own cookbook to do this.

To create a new cookbook we again use knife but first we must create a separate directory to store our custom cookbooks. This must be done as some Chef commands that manage cookbooks can delete cookbooks that have not been downloaded. It also helps organise your cookbooks clearly. So from the chef directory run

mkdir site_cookbooks

Then we must tell Chef about the new cookbook directory by editing the Vagrantfile to describe cookbook locations relative to the Vagrantfile:

chef.cookbooks_path = ["../chef/cookbooks", "../chef/site_cookbooks"]

Now instruct knife to create an empty cookbook for us (run from the chef directory)

knife cookbook create dbsetup -o site_cookbooks

If you look inside the site_cookbooks directory you will see a dbsetup cookbook that is far from empty. Fortunately we don't need to worry about most of this structure for the moment, we just need to edit the default recipe (site_cookbooks/dbsetup/recipes/default.rb):

execute "create_db" do
  command "mysql -uroot -p#{node[:mysql][:server_root_password]} -e "CREATE DATABASE IF NOT EXISTS #{node[:mydomain][:db_name]}""
end

execute "create_user" do
  command "mysql -uroot -p#{node[:mysql][:server_root_password]} -e "grant all on #{node[:mydomain][:db_name]}.* to '#{node[:mydomain][:db_user]}'@'localhost' identified by '#{node[:mydomain][:db_pass]}'""
end

This will instruct chef to run mysql commands to create a database and then a database user. This operation requires root permissions but we can fetch that here from the config we defined earlier. Note the database name, username and password are also pulled from the config. So best define that back in the Vagrantfile:

chef.json = {
  "mysql" => {
    "server_root_password" => "secure_password"
  },
  "mydomain" => {
    "db_name" => "app_db",
    "db_user" => "app_user",
    "db_pass" => "app_pwd"
  }
}

Also tell Chef to use the new cookbook:

chef.add_recipe "dbsetup::default"

And kick it off again with (from vagrant directory)

vagrant provision

This time you might see some unpleasant red text in the Chef output:

Shared folders that Chef requires are missing on the virtual machine.

As we created the new cookbook while the VM was running the directory could not be mounted. No problem, we can switch off and switch on again to fix:

vagrant reload --provision

vagrant ssh

mysql --user="app_user" --password="app_pwd"

mysql> show databases;

+--------------------+
| Database           |
+--------------------+
| information_schema |
| app_db             |
+--------------------+

2 rows in set (0.00 sec)

Excellent! We can login with our new user and that user can see the new database.

Using Chef to restore a database from a dump file

If only we could fill that database with data from a database dump so our VM has data to work with out of the box. Again Chef makes that pretty simple. First we need to generate the dumpfile. As we 'backup like a boss' around here use this command on whichever server contains the populated database: (substituting your db credentials):

mysqldump -p -u  | gzip > backup.sql.gz

Copy this file to site_cookbooks/dbsetup/files

Now add lines in the recipe to copy this file to the VM and restore the db (site_cookbooks/dbsetup/recipes/default.rb)

cookbook_file "backup.sql.gz" do
  path "/tmp/backup.sql.gz"
  action :create_if_missing
end

execute "import_db" do
  command "gunzip < /tmp/backup.sql.gz | mysql -uroot -p#{node[:mysql][:server_root_password]} #{node[:mydomain][:db_name]}"
end

And again get chef to work

vagrant provision

Now inspect the database to check your data is there.

vagrant ssh

mysql --user="app_user" --password="app_pwd" app_db

mysql> show tables;

+-----------------------+
| Tables_in_app_db      |
+-----------------------+
| wp_commentmeta        |
| wp_comments           |
| wp_links              |
| wp_options            |
| wp_postmeta           |
| wp_posts              |
| wp_term_relationships |
| wp_term_taxonomy      |
| wp_terms              |
| wp_usermeta           |
| wp_users              |
+-----------------------+

11 rows in set (0.00 sec)

I really like this setup. With our custom cookbook added to version control we have a setup that can from nothing create a VM, install core applications and also populate and configure our database. These methods can be used to setup Apache, PHP or whatever stack you require. This setup is also going to payoff for our EC2 server that we setup in the previous article. As we have done all the hard work creating the cookbook we only need to update the EC2 Vagrantfile with the cookbooks to run and the config. What's nice here is we can use the config to set different parameters for the different environments when required.

Here's the completed Vagrantfile for the local VM (mydomain/vagrant_local/Vagrantfile)

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

    config.vm.box = "ubuntu/trusty64"

    config.vm.network "private_network", ip: "192.168.56.210"

    # Install chef on the VM if required using the vagrant omnibus plugin
    config.omnibus.chef_version = :latest

    # Use chef to provision this machine
    config.vm.provision "chef_solo" do |chef|

        # Cookbooks directory path relative to this file
        chef.cookbooks_path = ["../chef/cookbooks", "../chef/site_cookbooks"]

        # Specify recipes we want 'cooked'. Format is ::
        chef.add_recipe "apt::default"
        chef.add_recipe "php::default"
        chef.add_recipe "php::module_mysql"
        chef.add_recipe "mysql::server"
        chef.add_recipe "mysql::client"
        chef.add_recipe "dbsetup::default"

        chef.json = {
            "mysql" => {
                "server_root_password" => "secure_password"
            },
            "mydomain" => {
                "db_name" => "app_db",
                "db_user" => "app_user",
                "db_pass" => "app_pwd"
            }
        }

    end
end

And here's the complete Vagrantfile for the remote EC2 server (mydomain/vagrant_ec2/Vagrantfile)

# https://github.com/mitchellh/vagrant-aws

# To start use vagrant up --provider=aws

Vagrant.configure("2") do |config|

    # Dummy box to work with Vagrant EC2 plugin
    config.vm.box = "dummy"

    config.vm.box_url = "https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box"

    config.vm.provider :aws do |aws, override|

        # https://blog.logentries.com/2014/03/devops-vagrant-with-aws-ec2-digital-ocean/

    aws.access_key_id = "AKIAIOSFODNN7EXAMPLE"
    aws.secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

        # http://cloud-images.ubuntu.com/locator/ec2/
        # us-east-1
        # trusty
        # 14.04LTS
        # amd64
        # hvm:ebs
        # ami-9aaa1cf2

        aws.ami = "ami-9aaa1cf2"
        aws.instance_type = "t2.micro"
        aws.region = "us-east-1"
        aws.availability_zone = "us-east-1b"

        # t2 instance types can only be launched to a VPC
        aws.security_groups = ["sg-1ab2c345"] # Replace with your security group
        aws.elastic_ip = "54.172.123.123" # Replace with your EIP
        aws.subnet_id = "subnet-123x456y" # Replace with your subnet id
        aws.keypair_name = "mykeypairname"
        override.ssh.username = "ubuntu"
        override.ssh.private_key_path = "/path/to/the/keypair/file.pem"

    end

    config.vm.provision "chef_solo" do |chef|

        # We have to put our home-made cookbooks in a separate directory
        chef.cookbooks_path = ["../chef/cookbooks", "../chef/site_cookbooks"]

        # Specify recipes we want 'cooked'. Format is ::
        chef.add_recipe "apt::default"
        chef.add_recipe "php::default"
        chef.add_recipe "php::module_mysql"
        chef.add_recipe "mysql::server"
        chef.add_recipe "mysql::client"
        chef.add_recipe "dbsetup::default"

        chef.json = {
            "mysql" => {
                "server_root_password" => "secure_password"
            },
            "mydomain" => {
                "db_name" => "app_db",
                "db_user" => "app_user",
                "db_pass" => "app_pwd"
            }
        }

    end
end

So there we have the basics of a project that can create a local VM and similar instance on EC2; provision both with applications via the same Chef setup and deploy databases. Now it's a matter of building on this structure to fill in the gaps and add the rest of the stack. For a start install a webserver and configure the virtual hosts files using Chef templates (maybe a future article). Also for production secure and then prepare methods to deploy codebases and databases. Happy dev-oping!