Not Dead Head

Side A

Monthly Archives: January 2011

Using Amazon EC2 … basics about setting up instances.

Todo: Keypair part is not right.

1 Amazon Elastic Compute Cloud (EC2)

1.1 Setup the environment

Step 0: have a linux or a mac box with java installed and setup … JAVA_HOME and stuff. You’re using Windows!? … sorry about that.

Step 1: Create your account at http://aws.amazon.com

  • To use the EC2 service you’ll need a Credit Card.
  • You need a telephone number. They (automated system) are going to call you to check for your existence.

Step 2: Install/Download the command line tools for working with EC2.

You can install the Ubuntu package or download them from the amazon site.

  • Ubunutu
$ sudo apt-get install ec2-api-tools
$ sudo apt-get install ec2-ami-tools
  • Download the tools from:
  1. Amazon EC2 API Tools
  2. Amazon EC2 AMI Tools

Step 3: Create a X.509 Certificate and download the private key and certificate to ~/.ec2

  • Go to “http://aws.amazon.com/” -> “Account2 -> “Security Credentials”
  • Under “Access Credentials” you can find three types of access credentials used to authenticate your requests to AWS services.
    • Go to “X.509 Certificates” and generate a new key/certificate pair.

This certificate is used to secure SOAP protocol requests to AWS service APIs.

Step 4: Setup environment variables

  • You can extend your .bashrc with this lines
#
# check for the right values

export EC2_HOME=`pwd`
export PATH=$PATH:$EC2_HOME/bin
export EC2_PRIVATE_KEY=`ls ~/.ec2/pk-*.pem`
export EC2_CERT=`ls ~/.ec2/cert-*.pem`

Make sure your JAVA_HOME environment variable is set. Don’t forget to source your new .bashrc!

1.2 AWS Regions and Availability Zones

Step 5: Test the installation

  • List the available regions using ec2-describe-regions
$ ec2-describe-regions
REGION	eu-west-1	ec2.eu-west-1.amazonaws.com
REGION	us-east-1	ec2.us-east-1.amazonaws.com
REGION	us-west-1	ec2.us-west-1.amazonaws.com
REGION	ap-southeast-1	ec2.ap-southeast-1.amazonaws.com

This is very important. Amazon has at this point in time 4 different locations for data centers spread around the world:

$ ec2-describe-regions
REGION	eu-west-1	ec2.eu-west-1.amazonaws.com
REGION	us-east-1	ec2.us-east-1.amazonaws.com
REGION	us-west-1	ec2.us-west-1.amazonaws.com
REGION	ap-southeast-1	ec2.ap-southeast-1.amazonaws.com

Amazon uses us-east-1 as default, unless you explicitly say which Regions and Availability Zones you want to use. To do this most commands have a parameter –region.

You can also set the default region using an environment variable:

export EC2_URL=https://ec2.eu-west-1.amazon.com

this will set the default region to EU for every command that is region aware.

1.3 Create an EC2 instance

Before we can do this some things need to be all setup. To be able to access an EC2 instance you’ll need:

  1. A key-pair for the SSH authentication.
  2. A set of firewall rules that allow you to connect into the instance.

Step 1: Create a Keypair.

This is a Keypair to use for ssh authentication. (pay attention to the –region)

$ ec2-add-keypair gsg-keypair > id_rsa-gsg-keypair
$ cat id_rsa-gsg-keypair
KEYPAIR	gsg-keypair	ee:27:58:80:a0:fc:ea:6f:94:ff:f1:fa:ce:e1:3c:12:27:2c:3e:89
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEArGmqlDdBs3vBovt6XMYoHLPmXDDlKvVDjuskoW09iJOk5TbKacn1mJC8+WPi
QNou5qzMac26pvTSLC7x2fg3z4+r1ZYnrvm8vKyPYfcU4YlEPHI4P1fhGk/uFtd+2t9Drly/Ja0Y
T09fxgVCsOGm/5FZQHD+n/q6mJRx0zVcnVKKnD8C3NM0q6KRsUGpelu+Egr+YffC0mkDr2ArhgJO
cveYVlITRZGA8VzTVPunKUQjLN1Ue+68SlerCXMnZ9Rvc0gk3T5t8QobQkYzsxbvSu+99AlPAs7G
1CGpd3OOiydRHTpj8Vj4NDYmJHZE8DaUBejo3ORpOAAMr3KVikE8FQIDAQABAoIBAHqiIiGQeBwI
/79ErlFE5Q0dbNHQOYDnuSjz2jcz81us4NHkue6rJxRghvnAcRL93fRXnoH3Qjc01jV1IrOOWWw6
XdfrGMjde8Cb4NSmoWSWdUGlYJq1kDhd+BSpLOLTgVmJQLC0wSDQbWf1H+2eY2FO6jPq60GxXn3e
wGOhhgGjot6DReIbbnSJBTwpP36JCcDArA5MGMrpG3ZCtlUlFof3NdwCmbrU4U16QeMi4yyW2coV
...
f/jQXG6iSl0V2jNdw6ERyUKNOVeWVmvIV/pjU4LTnL0yL/N87eWnpX5osFLbw/PssPXOpR2kVwOS
QFAD21AApIqF3z+Dy+cvcFqaQWraiMdIAG+rJmTw7U8xrl9Yp2Oklno/V725yPjvEPw=
-----END RSA PRIVATE KEY-----

You will need to take the private key that it creates, save it to a file called i.e. id_rsa-gsg-keypair, then set the right permissions (600 or rw- — —).

You can check which Keypairs are available. In this case I created a set of keys using the --region eu-west-1 parameter and one without it.

$ ec2-describe-keypairs --region eu-west-1
KEYPAIR	My_Key	ef:3f:eb:ac:8d:b9:be:61:5a:fe:92:c9:ff:43:0a:02:61:77:d1:1a
$ ec2-describe-keypairs --region us-west-1
$ ec2-describe-keypairs --region us-east-1
KEYPAIR	gsg-keypair	cf:31:1b:33:d5:5a:a1:c2:85:cc:f9:6c:64:22:2a:be:70:cb:9f:c1
$ ec2-describe-keypairs
KEYPAIR	gsg-keypair	cf:31:1b:33:d5:5a:a1:c2:85:cc:f9:6c:64:22:2a:be:70:cb:9f:c1

Notice that without the region parameter or the EC2_URL environment variable, the commands always default to the us-east-1 zone.  That means that you’ll never see the information of the eu-west-1 elements, if you don’t use the –region parameter or set the environment variable.

Step 2: Create an EC2 instance.

Let’s list some of the available AMIs (Amazon Machine Images) in eu-west-1:

$ ec2-describe-images --region eu-west-1 -a  | grep -ie "rightscale-eu.*centos_5.4_"
IMAGE	ami-efe4cf9b	rightscale-eu/CentOS_5.4_i386_v4.4.10.manifest.xml	411009282317	available	public		i386	machine	aki-7e0d250a	ari-7d0d2509		instance-store
IMAGE	ami-ddf8d3a9	rightscale-eu/CentOS_5.4_i386_v5.1.1_Alpha.manifest.xml	411009282317	available	public		i386	machine	aki-7e0d250a	ari-7d0d2509		instance-store
IMAGE	ami-ebe4cf9f	rightscale-eu/CentOS_5.4_x64_v4.4.10.manifest.xml	411009282317	available	public		x86_64	machine	aki-780d250c	ari-7f0d250b		instance-store
IMAGE	ami-37f8d343	rightscale-eu/CentOS_5.4_x64_v5.1.1_Alpha.manifest.xml	411009282317	available	public		x86_64	machine	aki-780d250c	ari-7f0d250b		instance-store

You can only create EC2 instances in a region using images available in that region.

Now you need to know what kind of machine you want to instantiate. There is list of profiles under http://aws.amazon.com/ec2/instance-types/. What you need to know is the API name.

Now lets create 2 EC2 instances:

  • using the rightscale-eu/CentOS_5.4_i386_v4.4.10.manifest.xml image
  • on a High-CPU Medium Instance
  • in Europe.
$ ec2-run-instances --instance-count 2 --region eu-west-1 --key My_Key --instance-type c1.medium  ami-efe4cf9b
RESERVATION	r-faf3de8d	975547918662	default
INSTANCE	i-fa10128d	ami-efe4cf9b	pending	My_Key	0	c1.medium	2010-12-23T14:55:13+0000	eu-west-1a	aki-7e0d250a	ari-7d0d2509	monitoring-disabled    instance-store
INSTANCE	i-f810128f	ami-efe4cf9b	pending	My_Key	1	c1.medium	2010-12-23T14:55:13+0000	eu-west-1a	aki-7e0d250a	ari-7d0d2509	monitoring-disabled    instance-store

Check that your machines where instantiated: (it will take a couple of minutes at most …)

$ ec2-describe-instances --region eu-west-1
RESERVATION	r-faf3de8d	975547918662	default
INSTANCE	i-fa10128d	ami-efe4cf9b	ec2-79-125-53-72.eu-west-1.compute.amazonaws.com	ip-10-85-214-22.eu-west-1.compute.internal	running	My_Key   0	c1.medium	2010-12-23T14:55:13+0000	eu-west-1a	aki-7e0d250a	ari-7d0d2509	monitoring-disabled	79.125.53.72	10.48.214.22	instance-store
INSTANCE	i-f810128f	ami-efe4cf9b	ec2-46-51-132-163.eu-west-1.compute.amazonaws.com	ip-10-85-193-229.eu-west-1.compute.internal	running	My_Key   1	c1.medium	2010-12-23T14:55:13+0000	eu-west-1a	aki-7e0d250a	ari-7d0d2509	monitoring-disabled	46.51.132.163	10.48.193.229	instance-store

1.4 Accessing the EC2 Instance

Now we have a couple of running instances. So how do we get to ride them! For this we need access to the instances.

The AWS system sets all instances behind a firewall and defines a set of rules which are hard bounded to the instances at creation time. These rules a called Security Groups. So to access an instance you need to define a security group which allows at least ssh to from a trusted source into the instance.

What do you mean by “hard bounded”? You cannot add Security Groups to an already created instance. You can only change the rule definition on the current Security Group.

You can define different Security Groups. If you don’t configure this at creation time all instances are bounded to the default security group.

$ ec2-describe-group  --region eu-west-1
GROUP	9755xx91862	default	default group
PERMISSION	9755xx91862	default	ALLOWS	all			FROM	USER	9755xx91862	GRPNAME	default

We need to define a rule that allows ssh (port 22) from a trusted host.

$ ec2-authorize --source-subnet xxx.xxx.xxx.xxx/32 --port-range 22 --protocol tcp --region eu-west-1 default
GROUP		default
PERMISSION		default	ALLOWS	tcp	22	22	FROM	CIDR	xxx.xxx.xxx.xxx/32
$ ec2-describe-group  --region eu-west-1
GROUP	9755xx91862	default	default group
PERMISSION	9755xx91862	default	ALLOWS	all			FROM	USER	9755xx91862	GRPNAME	default
PERMISSION	9755xx91862	default	ALLOWS	tcp	22	22	FROM	CIDR	xxx.xxx.xxx.xxx/32

Now we can access our instantiated EC2 servers:

$ ec2-describe-instances --region eu-west-1
...
INSTANCE	i-f81x128f	ami-efexcf9b	ec2-46-xx-132-163.eu-west-1.compute.amazonaws.com	ip-10-xx-193-229.eu-west-1.compute.internal	running	My_Key	1	c1.medium	2010-12-23T14:55:13+0000	eu-west-1a	aki-7e0dxx0a	ari-7d0dxx09	monitoring-disabled	46.xx.132.163	10.xx.193.229	instance-store

$ ssh -i key_pair/My_Key.pem root@ec2-46-xx-132-163.eu-west-1.compute.amazonaws.com
The authenticity of host 'ec2-46-xx-132-163.eu-west-1.compute.amazonaws.com (46.xx.132.163)' can't be established.
RSA key fingerprint is 24:fc:75:8a:...:7e:f3:11:e4:e9.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-46-xx-132-163.eu-west-1.compute.amazonaws.com,46.xx.132.163' (RSA) to the list of known hosts.
     ___   _        __   __   ____            __
    / _ \ (_)___ _ / /  / /_ / __/____ ___ _ / /___
   / , _// // _ `// _ \/ __/_\ \ / __// _ `// // -_)
  /_/|_|/_/ \_, //_//_/\__//___/ \__/ \_,_//_/ \__/
           /___/

Welcome to a public Amazon EC2 image brought to you by RightScale!

********************************************************************
********************************************************************
***       Your EC2 Instance is now operational.                  ***
***       All of the configuration has completed.                ***
***       Please check /var/log/install for details.             ***
********************************************************************
********************************************************************
[root@ip-10-xx-193-229 ~]#

1.5 Shutting down the instances

After you use your EC2 instances you can terminate them and stop the billing process.

$ ec2-describe-instances --region eu-west-1
RESERVATION	r-faf3de8d	9755xx91862	default
INSTANCE	i-fa10128d	ami-efexcf9b	ec2-79-xx-53-72.eu-west-1.compute.amazonaws.com	ip-10-xx-214-22.eu-west-1.compute.internal	running	My_Key	0		c1.medium	2010-12-23T14:55:13+0000	eu-west-1a	aki-7e0d250a	ari-7d0d2509		monitoring-disabled	79.xx.53.72	10.xx.214.22			instance-store
INSTANCE	i-f810128f	ami-efexf9b	ec2-46-xx-132-163.eu-west-1.compute.amazonaws.com	ip-10-xx-193-229.eu-west-1.compute.internal	running	My_Key	1		c1.medium	2010-12-23T14:55:13+0000	eu-west-1a	aki-7e0d250a	ari-7d0d2509		monitoring-disabled	46.xx.132.163	10.xx.193.229			instance-store	

$ ec2-terminate-instances --region eu-west-1 i-f810128f i-fa10128d
INSTANCE	i-f810128f	running	shutting-down
INSTANCE	i-fa10128d	running	shutting-down

$ ec2-describe-instances --region eu-west-1
RESERVATION	r-faf3de8d	975547918662	default
INSTANCE	i-fa10128d	 ami-efexcf9b	terminated	 My_Key	0	c1.medium	2010-12-23T14:55:13+0000	eu-west-1a	aki-7e0d250a	ari-7d0d2509	monitoring-disabled	instance-store
INSTANCE	i-f810128f	 ami-efexcf9b	terminated	 My_Key	1	c1.medium	2010-12-23T14:55:13+0000	eu-west-1a	aki-7e0d250a	ari-7d0d2509	monitoring-disabled	instance-store

Terminated instances will be deleted and you won’t need to pay anymore. This is different from stopped instances, they are kept.

%d bloggers like this: