Running LAMP, Ubuntu and Symfony on Amazon's EC2 - It's Awesome! (Part 2)

Posted by ryan in ec2, symfony, ubuntu

Woo! You're reading part 2 of a trilogy:
   - Part 1 - Running LAMP, Ubuntu...
   - Part 3 - Running LAMP, Ubuntu... (in progress)

cloud computing simpons - no relation
totally unrelated

On our way to a fully-functioning cloud server

In Part 1 of this series, we signed up for our Amazon AWS account, got our local system setup with our AWS private key and certificate, booted an instance, and explored Amazon's EC2 console. Now things get interesting. Today we'll tackle the following:

Assign an elastic ip address to your instance

Each time you start up your instance, you're assigned a different internal Amazon host / ip address that you can use to ssh into your system. Things get dicey, however, when you think about your DNS. If your instance needed to be restarted, then you'd also have to update your DNS A record for your new ip address. Since this can take many hours to propogate, the whole world could be looking at the old server way longer than you intended.

Finding the ip address assigned to you. After booting up an instance, Amazon always assigns it an ip address. You can bind this to an, existing static ip address (read below), but you may need to login to your new instance before doing that. To find out your ip address / host:
# ec2-describe-instances
The response will show all your current instances. Look for the ec2-##-###-###-##.compute-1.amazonaws.com. You can either use that as a host or use the ip address directly, which is simply the above numbers represented by # signs.

Enter the idea of "elastic ip" addresses. In EC2, you can "allocate" ip addresses. This means that you request a static ip address from Amazon which then stays with your EC2 account forever (unless you don't need it anymore and "release" it). Once you have an ip address, you can "Associate" it with whatever running instance you have and that ip address will begin resolving to that instance instantly. You may also now safely use this ip address for your DNS A record without fear of it needing to be changed later.

Elastic ip addresses in EC2

"Allocate" an ip address and then "Associate" it with your instance

  1. Login to your EC2 console: http://console.aws.amazon.com/
  2. Click Elastic IPS on the left navigation
  3. Click Allocate New Address near the top
  4. Click Associate near the top
  5. On the window that pops up, select the correct instance id and click Associate

You should now be able to ssh into your instance directly via the elastic ip address.

Are these elastic IP addresses free? Well, yes and no. As long as your elastic ip address is associated with one of your running instances it's completely free. Amazon charges only for elastic ip addresses that have been allocated to your account but which aren't associated with an active instance.

Customize your server and save it as an image (AMI) for use later

As explained in part 1 of this series, the server you've instantiated acts as a static entity. Yes you can make all of the changes and customizations you want to it just like any other server. However, as soon as you shutdown that instance, all of those changes are lost forever. The solution is to save your server setup as an image (called an Amazon Machine Image or AMI), which can then be used later to instantiate new instances. This actualy forces you into the very responsible habit of always having an image of your exact server setup. From here you can very easily restart your instance or launch several identical servers and load balance between them.

While this section is really more appropriate for after we get our server setup with LAMP, I've placed it here because of its importance. Always be sure to save a new AMI after materially changing your server. If you don't, those changes will die whenever you inevitablly need to restart your server.

  1. ssh into your server
  2. Shutdown Apache and MySQL (we haven't added these yet, but for future reference, you'll need to stop them)
    # /etc/init.d/apache2 stop
    # /etc/init.d/mysql stop
  3. Unmount your EBS (again, we haven't mouned our EBS, but use this for future reference)
    # sudo umount /vol
  4. Bundle your image. This will create an image of your instance and store it locally in /mnt
    # sudo ec2-bundle-vol -d /mnt -k ~ubuntu/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem -c cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem -u 3634-9027-7607 -s 1536
  5. You'll now upload your image from your cloud up to your S3 account. Before you do this step, you'll need to create a "bucket" in s3 where your image will live. Once that's setup, you'll need your amazon access key id and secret access key, which you can getfrom the "Access Identifiers" section of your aws account area (http://aws.amazon.com/account/)
    # ec2-upload-bundle -b your-s3-bucket-name -m /mnt/image.manifest.xml -a YOUR_ACCESS_KEY -s YOUR_SECRET_ACCESS_KEY
  6. In your local terminal (meaning NOT in your cloud terminal, but on your local machine), run the following command, replacing your-s3-bucket-name with the bucket you created and used in the previous step
    # ec2-register your-s3-bucket-name/image.manifest.xml
    The above will "register" your saved image with Amazon, so that you can find and use it later to boot instances. The return from this call will be the unique identifier for this image, which you'll use when booting instances from it. The response should look something like this:
    response: IMAGE	ami-a12345bc
  7. Now that we've finished the imaging of your instance, we've just gotta put the pieces back together
    # sudo mount -a
    # sudo /etc/init.d/apache2 start
    # sudo /etc/init.d/mysql start

Attach and mount an EBS (Elastic Block Store) and point your MySQL databases at it

As explained above, nothing on your actual server gets saved unless you explicitly create an image from it. In this way, your server really functions like a read only device: only save things to it that you don't want to keep.

So, where do we keep our dynamic content, our databases, our uploaded assets? Amazon's Elastic Block Store (EBS) is basically an external hard drive that you mount to your static instance. You can have as many EBS volumes as you need, they can be nearly any size and, most important, EBS volumes live forever and ever. Let's hook up an EBS and get MySQL running on it. We'll start by bringing our EBS into existence.

Turn your EBS on and "plug it in"
  • Login to your EC2 console: http://aws.amazon.com/console/
  • Click "Instances" on the left, then select the instance you currently have running. In the information that displays below, look for the "Zone" of your instance (e.g. us-east-1b). You'll need to create your EBS in the same zone as your instance, so keep that info handy.
  • Click on "Volumes" on the left and then click "Create Volume" on the resulting screen.
  • Set the size (see note below related to changing your size later) and choose the same Availability Zone as your instance. Leave the Snapshot blank and click Create. The volume will take just a little while to process.
  • Once the instance is up and running, select the volume and click "Attach Volume". Basically, you've turned your hard drive on, but you still need to plug it into your instance. On the resulting screen, make sure the correct instance is selected and choose /dev/sdh as your Device. Click attach. Nice :)
Changing the size of your EBS
While you can't technically increase the size of your EBS after creation, it's actually quite easy in practice. From the console, simply choose your existing volume and click "Create Snapshot". Once this backup is complete, create a new EBS setting the Size to your new desired size and Snapshot to the name of the snapshot your just created. Now that you have two identical EBS volumes (except one is bigger than the other), simply detach the old volume (you'll need to unmount it first) and attach the new volume. Remount and celebrate. You're a cloud king.

Now that our EBS is alive and "plugged into" your instance, we need to mount it and get mysql put in the right spot.

Mount your EBS and put MySQL on it
  • Shell into your instance
  • First, we'll make sure we're updated and then get xfs and mysql installed. We'll use xfs to format our EBS volume.
    # sudo apt-get update && sudo apt-get upgrade -y
    # sudo apt-get install -y xfsprogs mysql-server
    # sudo modprobe xfs
    # sudo mkfs.xfs /dev/sdh
    # echo "/dev/sdh /vol xfs noatime 0 0" | sudo tee -a /etc/fstab
  • Now we can simply mount our EBS at /vol
    # sudo mkdir /vol
    # sudo mount /vol
  • Next, move the existing mysql files to the EBS volume then point MySQL to the correct database files on the EBS volume by using mount bind.
  • # sudo mkdir /vol/etc /vol/lib /vol/log
    # sudo mv /etc/mysql     /vol/etc/
    # sudo mv /var/lib/mysql /vol/lib/
    # sudo mv /var/log/mysql /vol/log/
    # sudo mkdir /etc/mysql
    # sudo mkdir /var/lib/mysql
    # sudo mkdir /var/log/mysql
    # echo "/vol/etc/mysql /etc/mysql     none bind" | sudo tee -a /etc/fstab
    # sudo mount /etc/mysql
    # echo "/vol/lib/mysql /var/lib/mysql none bind" | sudo tee -a /etc/fstab
    # sudo mount /var/lib/mysql
    # echo "/vol/log/mysql /var/log/mysql none bind" | sudo tee -a /etc/fstab
    # sudo mount /var/log/mysql
    # sudo /etc/init.d/mysql start
Wait, what the hell just happened?

Ok, let's regroup after the shell-command barf. An EBS volume is nothing more than a glorified external hard drive. All of that setup equates to 4 basic steps

Setup Apache

With the hard work behind us, let's setup apache. Setting up apache is nothing special on the cloud. The only thing you have to do is make sure that everything ends up on your EBS (/vol), which includes your actual site files, as well as any logs that you might want to reference later.

# sudo apt-get install php5-curl php5-cli php5-tidy php5-xsl apache2 php5-mysql php5-syck

I'll leave the rest up to you - there are lot's of resources on how to get apache up and rocking.

Sync up a symfony project and get it rocking

If you've gotten this far and gotten your vhost setup for symfony, you're basically done. At this point, your cloud offers no additional differences from any other VPS that you've worked on before.

If you use Symfony's sync task, however, you'll have one more obstacle to jump through. Symfony's project:deploy task really wasn't built to shell into servers using a public key. No worries, I've already suffered through this and created a replacement task for use in Symfony 1.2. Just be sure to add a keypair value to your properties.ini file:


Final Thoughts - What's left?

Ok, so this wasn't really an article on symfony, except for the last little task I threw in on the end. In general though, I had a hell of a time getting the cloud up. In the end, I'm a better person for it and I've been very happy with how well the cloud has been running. Being hooked up with Amazon also means that I can easily back up my EBS volumes to S3 as well as start up new instances just to play around with (e.g., try out performance tweaks away from your main server). Above all else, clouds a hella-fun but expensive unless you really need to flex something.

What's left to consider is a) cost and b) benchmarks. I'm thinking it'll be a trilogy.

Thanks for the shares!
  • StumbleUpon
  • Sphinn
  • del.icio.us
  • Facebook
  • TwitThis
  • Google
  • Reddit
  • Digg
  • MisterWong
Posted by Paco on 2009-08-26
Thanks for sharing your experience. I hear a lot about cloud computing, but in so general terms. Your article at last showcases a hands-on that makes cc understanble & tangible at last !-)
Posted by Kevin on 2009-09-11
Looks like the link to your ec2-project-deploy-cloud-task code isn't working...

Good article, btw!
Posted by Alex Stoneham on 2009-09-14

Great Article.

You should also include how to set up symfony under control (continuous integration) and staging in a part three.

Cool article, I can vouch for your steps, they are amlost the same as how I did it. I wish I had this article a few months ago.
Posted by Ryan on 2009-09-20

I feel stupid :) - very dumb mistake on my part. The download link for the task should be working now - thanks for the head's up!
Posted by Ryan on 2009-09-20
@Alex Stoneham

Glad to hear that we were on the same page. I think that covering symfony under control is a great idea - I have set this up before, and it can be a fairly confusing process. I'll see if I can't throw it in there.
Posted by Dennis Gearon on 2009-10-13
I will be really interested in the costing/analysis. Clouds are pushed as a way to run little stuff and turn it into big stuff later, easily. But I'm paying ~$20/mos for a decent Postgres Running site, and they say that they will be able to push it up to to full virtualized performance on some heavy iron, painlessly, anytime I want, and in several cost stages. ($they='a2hosting:very_nice_guys). So I will be looking to the cloud when I need to go past that, or sooner to make the transition easier. Based on costing of course.
Posted by Kevin on 2010-01-01
FYI, if anyone is trying to use ec2projectdeploycloudtask in Symfony 1.4, you have to change "getFilesystem()->sh(" to "getFilesystem()->execute(" on the last line in ec2ProjectDeployCloudTask.class.php.

Symfony changed a lot of stuff in 1.3/1.4, including removing sfFileSystem::sh().

Posted by rp on 2010-02-04
excellent post! very informative, i have been investigating ec2 hosting options and found that it as very difficult to get an idea of hosting costs. I understand you mentioned that there are fixed costs (instances) and then variable costs (bandwidth + disk space).

The usage log that you can download doesn't make much sense either. Do you know of any tools that can be used to convert them into something more meaningful?
Posted by Ryan on 2010-02-05
Hey rp-

I can't say anything about that tool (I don't know it), but I can give you a general idea about the costs. For me, almost 100% of my costs are for the instance hours themselves. The bandwidth and disk space end up being SUPER cheap compared with the instance hours unless you're doing something outrageous. For me, I'd have to buy more instance hours long before my traffic took my bandwidth up to any level worth worrying about.
Posted by rp on 2010-02-06
hey Ryan, thanks for the notification, I guess that's a fair assumption and that's kinda what we boiled it down to, since we were planning to run steady state websites, reserved instances turned out to be the cheapest option and like you said bandwidth/space was cheap.

So just out of curiosity is patching and maintaining servers an overhead that you don't have to deal with managed hosting or the updates available on ubuntu is good enough.
Posted by JamieL on 2010-03-11
Ryan, great article thanks. You've saved a lot of headaches, I'm sure.

May want to re-make the AMI after the EBS integration to save those steps each new instance.
Posted by Shane on 2010-06-15
Great tutorial - the best I've found. Are you still planning on completing the 3rd part of the series? Eager to learn more... Thanks!
Posted by Shane on 2010-06-15
Great tutorial - the best I've found. Are you still planning on completing the 3rd part of the series? Eager to learn more... Thanks!
Posted by Andu on 2010-11-10
I guess the AMI can't be installed by anyone after putting it in the bucket - I imagine it's readable, and it would be very easy for someone to find the URL. I'm 99% sure, actually, because that AMI also contains some very sensitive info (like DB passwords), not to mention the entire webapp.
Posted by Damien on 2010-11-25
Thanks for the detailed article on a useful subject. Still planning on completing the trilogy?
Posted by Ryan on 2010-11-29
Right now I think this'll remain as a two-parter, but I'd love for someone else to pick it up. Also, my impression is that a lot has changed with EC2 since publishing this - an updated post is probably needed.
Posted by Joshua on 2010-12-29
Really good article, I keep finding myself coming back to skim over it again. I recently started using ec2 and was wondering if you have used symfony with more advanced configurations such as multiple mysql databases and/or memcached servers?
Posted by free drive recovery on 2011-11-04
Thnaks for this excellent notification, It helps in easily back up EBS volumes to S3 as well as start up new instances.
Posted by PK on 2011-12-03
Thanks for these articles, they must have saved me countless hours!