Where did my Rsync output go?

For those of you who are fans of symfony's project:deploy (sync) task, you'll undoubtedly notice the lack of rsync output in Symfony 1.3. However, the output isn't actually missing - it's just hidden by default. Use the -t (trace) option to uncover it.

  ./symfony project:deploy -t production

As an added bonus, the output is now rendered to the command line in real time due to symfony's new handling of shell commands. Nice going guys!

Woo! You're reading part 1 of a trilogy:
- Part 2 - Running LAMP, Ubuntu...
- Part 3 - Running LAMP, Ubuntu... (in progress)

Quickie Introduction to EC2

For those of you that get bored with paragraph-style introductions to blog entries, here's a bulleted list for you:

• End result feels like a VPS, except you can change the specs of your VPS in seconds.
• Ability to launch a bunch of identical servers in seconds:
  • Kickass deployment: bring up a 2nd, identical server, push your new site version there, test vigorously, point amazon to use your new instance, shutdown the old instance.
  • Play around: Start up an identical server except with faster hardware. Benchmark and playaround. When you're done playing, shutdown the instance - you're only charged by the hour
• By definition, you full server image is always saved to S3, so you have an exact copy of your setup in a very safe place
• Backup your dynamic data (databases, uploads dir) to S3 in seconds
• Basically, you can do a ton of cool stuff - I'm a believer!

The Setup

OS	Ubuntu
Web server	Apache 2.2 with MySQL & PHP5
End Use	Symfony 1.2.8 Web Application
Mail Server	Nope - this is being outsourced to google, no need for the headache or overhead

Signing Up & Downloading your Private Key and Certificate files

To get started, you'll need both an EC2 account and an S3 account.

• EC2 - The actual cloud computing account
• S3   - Storage, used to save images of your server

After you signup, you'll need to download 2 files: a Private Key and a Certificate file. These are basically the keys to the mansion. Here's a great resource on this simple step:

Whenever you start an instance [turn your "server" on], you must select a fully-functional image that you want that server to use. Fortunately, the ec2 community has published thousands of images for all operating systems and setups that you will use to boot from initially. From there you can customize your server exactly as you need it. When you're done, you need to save your new, customized image somewhere so you can boot from it. This is done easily in EC2 and stores the image on your S3 account. Once there, you can boot any new instances from your full-customized server image.

EC2 Tools

Basically, there are 2 ways to screw around with EC2:

• Command line, via the awesome ec2-tools
• Gui web interface, via the awesome Amazon console

In reality, you'll need to use both tools. First, let's setup your local computer to use the EC2 tools

sudo apt-get install ec2-api-tools
sudo apt-get install openjdk-6-jdk

Remember that Private Key and that Certificate file you downloaded from Amazon earlier? You'll need those here. Basically, the ec2 tools look for those files in a certain shell variable. Also, you'll need another variable for JAVA, which ec2 uses. Paste the following into your ~/.bashrc file:

export EC2_PRIVATE_KEY=$HOME/<where your private key is>/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem
export EC2_CERT=$HOME/<where your certificate is>/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/

Starting up your First Instance

Starting an "instance" basically means that you push the "power" button on one of Amazon's virtual servers. The cool part is that in addition to turning a machine on, you also specify exactly what operating system and setup you want that instance to have. There are literally thousands that you can use as a starting point, which you'll customize afterwards.

Amazon is loaded with images (known as AMI's) ready to be used to start up your machine. To view them, use the ec2-describe-images command. As this will return MANY results, I grep the results here for only ubuntu AMIs.

ec2-describe-images -a | grep ubuntu

...
IMAGE	ami-5059be39	canonical-cloud-us/ubuntu-intrepid-20090422-i386.manifest.xml	099720109477	available	public		i386	machine	aki-714daa18	ari-6a5bbc03

I chose the above image as it's the latest (at this time) AMI directly created by Canonical, the creators of Ubuntu itself. The important figure here is the ami-######## string, which identifies the AMI. Let's get a machine booted up with that image. From the command line:

# create a keypair, this'll be used all over the place for access
ec2-add-keypair ec2-keypair > ec2-keypair.pem
chmod 600 ec2-keypair.pem
ec2-run-instances ami-5059be39 -k ec2-keypair
ec2-describe-instances 

That's it! You're EC2 instance is now botting up. The i-XXXXXXXX portion is the id of your instance, which is used to shut it down, among other things. You can continue to run the ec2-describe-instances to see the status of your instance. It usually takes a few minutes for the machine to fully start up, so wait until it says that it is "running".

As soon as you boot up an instance, you're being billed. To shutdown an instance, use the i-XXXXXXXX id from the ec2-describe-instances command.

ec2-terminate-instances i-XXXXXXXX

EC2 Console - GUI goodness and a flex of the power of EC2

EC2 comes with a fantastic gui control panel that allows you to boot up instances, manage your AMI's (images), and do a lot of other things that we'll touch on later. http://console.aws.amazon.com/

Here's a quick run-down of the more important links along the left side:

• INSTANCES -> Instances:
  Displays all the actual booted up instances that you have with EC2
• IMAGES -> AMIs (Amazon Machine Images):
  Allows you to browse all the images that are available (including ones that we'll create later for our own use). You can start up an instance with any images found here at any time
• ELASTIC BLOCK STORE -> Volumes
  These are basically hard drives that you'll mount to your server and is used for any dynamic data (databases, logging, web sites).
• NETWORKING & SECURITY -> Elastic IPs
  Essentially a static ip you'll bind to your instance later
• NETWORKING & SECURITY -> Security Groups
  Each instance belongs to one or more security groups (your instance belongs to the "default" security group by default). The security groups define which ports are open to the public. By default, your instance won't be open on any ports, but we'll open them up one by one as necessary

Your server is not persistent. In other words, if I boot up my instance, create a file, then shutdown my instance, that file is gone forever. To save changes to your server, you create your own image (AMI) which you can then boot from later. So, other than software changes, apache configurations, etc, your server should stand as a static begin.

What about databases, upload directories and log files? This is where the EBS volume comes in. An EBS volume is essentially a hard drive that exists as a separate entity from your instance, which you then mount to your instance. An EBS is persistent, meaning that any file you copy into it will remain after shutting down your instance. You can even connect to an EBS from multiple instances. This is where everything dynamic - particularly your databases - will live.

SSH into your instance

SSH'ing into your instance is simple. First, however, we need to open up our server on port 22. We do this by telling the "default" security group (read above about Security Groups) to allow connections from port 22. You'll also need to open up other ports later - specifically port 80 when you get your webserver rocking.

ec2-authorize default -p 22

To ssh in, we need to know the internal host that amazon has assigned to us. The host is that one that starts with ec2- and ends with .amazonaws.com. By the way, this will change each time you shutdown and reboot your image. No fear, later we'll bind a static ip address to the instances to solve this problem.

ec2-describe-instances
...
INSTANCE	i-########	ami-a37190ca	ec2-##-##-###-###.compute-#.amazonaws.com	...

While most EC2 images are setup so that you login as root, the one we installed (made by Cononical) uses the user "ubuntu" (you can then sudo easily once you're in). We'll use the ec2-keypair.pem that we created above to authenticate into the server.

ssh -i ec2-keypair.pem ubuntu@ec2-##-##-###-###.compute-#.amazonaws.com

You should now be in your server. As the welcome message explains, you can get a bunch of things setup on your server fairly easily by running the command sudo tasksel --section server. I selected the following:

• Basic Ubuntu server
• LAMP server
• OpenSSH server

Scp your private key and certificate to your cloud

In order to do things such as creating an image from your server setup, you're going to need that pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem and cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem files from earlier on your cloud. To get them up there, we can simply use scp

scp -i /path/to/ec2-keypair.pem /path/to/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem /path/to/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem ubuntu@ec2-##-##-###-###.compute-#.amazonaws.com:

We'll come back and use those in part 2 of this article, so just let them sit there for now.

Where we are and what we're missing still

What we did:

• Signed up for EC2
• Setup ec2 tools
• Picked out an image (AMI) and booted up an instance with it)
• Logged into your new instance

What's still to come:

• Currently, if you shutdown your new instance, any server changes you made are lost. We'll need to customize our server and then save our own AMI. This will take a snapshot of the server (with our customizations) which we can boot from later on.
• Create an EBS volume (a hard disk that mounts to your server, but whos data persists after you shutdown your instance) and move all of the dynamic data to it (Mysql, webserver, logs, etc)
• Associate an elastic ip address with your instance so you can setup your DNS to point at that ip address
• Get a Symfony project rocking on your server
• Explore potential tripping points, like how to properly rsync your symfony project to your EC2 instance