HBase EC2
This collection of scripts allows you to run HBase clusters on Amazon.com's Elastic Compute Cloud (EC2) service described at:
http://aws.amazon.com/ec2
To get help, type the following in a shell:
bin/hbase-ec2
You need both the EC2 API and AMI tools
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=368&categoryID=88
installed and on the path.
When setting up keypairs on EC2, be sure to name your keypair as 'root'.
Quick Start:
1) Download and unzip the EC2 AMI and API tools zipfiles.
For Ubuntu, "apt-get install ec2-ami-tools ec2-api-tools".
2) Put the tools on the path and set EC2_HOME in the environment to point to
the top level directory of the API tools.
3) Configure src/contrib/ec2/bin/hbase-ec2-env.sh
Fill in AWS_ACCOUNT_ID with your EC2 account number.
Fill in AWS_ACCESS_KEY_ID with your EC2 access key.
Fill in AWS_SECRET_ACCESS_KEY with your EC2 secret access key.
Fill in EC2_PRIVATE_KEY with the location of your AWS private key file --
must begin with 'pk' and end with '.pem'.
Fill in EC2_CERT with the location of your AWS certificate -- must begin
with 'cert' and end with '.pem'.
Make sure the private part of your AWS SSH keypair exists in the same
directory as EC2_PRIVATE_KEY with the name id_rsa_root. Also, insure that
the permissions on the private key file are 600 (ONLY owner readable/
writable).
4) ./bin/hbase-ec2 launch-cluster <name> <nr-zoos> <nr-slaves>, e.g
./bin/hbase-ec2 launch-cluster testcluster 3 3
5) Once the above command has finished without error, ./bin/hbase-ec2 login
<name>, e.g.
./bin/hbase-ec2 login testcluster
6) Check that the cluster is up and functional:
hbase shell
> status 'simple'
You should see something like:
3 live servers
domU-12-31-39-09-75-11.compute-1.internal:60020 1258653694915
requests=0, regions=1, usedHeap=29, maxHeap=987
domU-12-31-39-01-AC-31.compute-1.internal:60020 1258653709041
requests=0, regions=1, usedHeap=29, maxHeap=987
domU-12-31-39-01-B0-91.compute-1.internal:60020 1258653706411
requests=0, regions=0, usedHeap=27, maxHeap=987
0 dead servers
Extra Packages:
It is possible to specify that extra packages be downloaded and installed on
demand when the master and slave instances start.
1. Set up a YUM repository. See: http://yum.baseurl.org/wiki/RepoCreate
2. Host the repository somewhere public. For example, build the
repository locally and then copy it up to an S3 bucket.
3. Create a YUM repository descriptor (.repo file). See:
http://yum.baseurl.org/wiki/RepoCreate
[myrepo]
name = MyRepo
baseurl = http://mybucket.s3.amazonaws.com/myrepo
enabled = 1
Upload the .repo file somewhere public, for example, in the root
directory of the repository,
mybucket.s3.amazonaws.com/myrepo/myrepo.repo
4. Configure hbase-ec2-env.sh thus:
EXTRA_PACKAGES="http://mybucket.s3.amazonaws.com/myrepo/myrepo.repo \
pkg1 pkg2 pkg3"
When the master and slave instances start, the .repo file will be added to
the Yum repository list and then Yum will be invoked to pull the packages
listed after the URL.
843bfd31b7