I guess I should start out with what Amazon S3 is I suppose… It’s a service that you can use to store reams of data, offsite, online, accessible anytime for a decent price. As of this writing the costs are:

Storage: $0.15 per GB-Month of storage used
Data Transfer: $0.10 per GB - all data transfer in

$0.18 per GB - first 10 TB / month data transfer out
$0.16 per GB - next 40 TB / month data transfer out
$0.13 per GB - data transfer out / month over 50 TB

see Amazon S3 for more details. If you use .mac, you can think of it like an iDisk on steroids - or a huge webdav server - but it’s not just one sever it’s a farm of servers with huge amounts of redundancy.

My goal here is to make a simple process that will automatically back up some of my web sites - for this example I am using this blog. What this process is going to do is backup the blogs MySQL database. This will just be a basic setup, but you can tweak it or add to it as you see fit.

The Plan

We are going to make some scripts that dump the database to a directory, and then sync that directory with the Amazon Service. The sync will work somewhat like rsync so if we want to add other files to the sync tree, it will just be a matter of adding them into sync area.

Amazon S3 Account

To start off, you’ll need an S3 account. You can sign up (and login) here. As you can tell, I like the Amazon Storage Service; however, their web site lacks in the usability department in my opinion. So if you get a bit confused trying to navigate the site, don’t worry you’re not alone.

After you go though the account setup process, you’ll need to get your access key id (like your username) and your secret access key (like your password). To find these hover over the Your Web Services Button button on the top right, and click AWS Access Identifiers (See shot below).

From that screen, on the right you should see your Access Key ID, and be able to show your Secret key. You’ll need these for the next steps.

Using S3 From the Desktop

You’ll likely want access to your S3 account from your desktop. If only to verify the backups are running. However, once you get this setup, see how well it works, and see how inexpensive it is you’ll probably start using it for other stuff (as I do :D).

For desktop access there are quite a few choices. I went the cheap route. I use cockpit. It’s a Java based application that lets you use your S3 account from the desktop. It has the feel of an FTP application. Jungle Disk is another one I hear is very good. It mounts the S3 account as if it’s a remote drive (on Mac) or a drive letter on windows (and it also works on Linux) - Jungle Disk is $20 USD.

I am going to use cockpit in this article, but they should all work about the same.

Start up cockpit. On Mac / Linux, run the file:

$ /Applications/jets3t-0.5.0/bin/cockpit.sh

(on windows it’s the cockpit.bat file)

When it first starts up there will be several ways to login. You can read up on the choices on the cockpit website, but for now we’ll use the Direct Login tab. On this tab is where you’ll put the keys you got from the S3 Account page (see image)

S3 uses things called buckets. You’ll see that term quite a bit when reading about S3. It’s just a place to store stuff. You can think of it as a drive letter (if you’re coming from a windows world), or a database table, or whatever. What we need to do here is make a bucket for our backups. I made one called server.robrohan.com - you can name yours anything you like. The shot below is of my server.robrohan.com bucket and a few backups that have already run.

To create a new bucket, click the gear icon on the left and select Create new bucket…. You’ll need at least one bucket to continue with this article.

Database Dump

Now that we have an Amazon S3 account setup, we can access it from our desktop, and we have one bucket ready to hold stuff we can start doing our backups. The first step in doing a backup of our MySQL database is to get a dump of the database. If you already have a process for this you can skip this section. (Note: This is a very simple way to get a backup of a MySQL database there are better, more efficient ways to do this.)

Here is a simple script to dump a database to a bziped file. You an call this script dbbackup.sh, and it is written so it can be called from cron. (This is pretty insecure, so be sure you’re ok with that or seek help in making it more secure if you need)

#!/bin/sh

# This file backs up a database to a backup dir
# that will be synced with s3

if [ $# != "3" ]; then
    echo “Usage:”
    echo ”      $0 [database] [username] [password]”
    exit 1
fi

DATE=`date +%d`
DB=$1
USER=$2
PASS=$3

BACKUPDIR=/home/my_user_name/backups
DBSTORE=${BACKUPDIR}/database

# create database backup
mysqldump –add-drop-table \
          -h localhost \
          –user=${USER} \
          –password=${PASS} \
          ${DB} \
          | bzip2 -c > ${DBSTORE}/${DB}_${DATE}.sql.bz2

Note the variable BACKUPDIR. This should be a directory you create somewhere on your server that you want to “sync” with the Amazon service. Meaning that anything you put in that directory will get backed up to S3. In this article we are just putting a database backup in the tree, but you could put server code, documents on the server, copies of configuration files, or whatever. The main point being anything in that directory will get copied to S3 (recursively - subdirectories as well).

The script is run like the following:

$ ./dbbackup.sh mydatabase myusername mypassword

And would produce the file:

/home/my_user_name/backups/database/mydatabase_[day_of_month].sql.bz2

In this process database backups will only go back a month. The first of the month will overwrite the last month’s first of the month backup. If you’d like to archive forever, just change the DATE variable to be `date +%y%m%d`. For my purposes, doing backups everyday is overkill.

Syncing with S3

So now the meat of the thing. There are, as in the desktop access, many server side scripts you can use to write data to S3. What this example is going to use is s3sync.rb. s3sync is a ruby script that acts much like rsync. There is also an outdated perl version, and even a bash version. You can try others if you’d like, but for this I’ll be using the ruby version. That, of course, means you’ll need ruby on the sever (not rails or anything, just ruby).

(If you don’t have ruby you can do apt-get install ruby, yum install ruby, ports install ruby, or whatever. It’s often very easy to get ruby on your *nix box)

Download and untar the s3sync file:

$ wget http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz
$ tar -zxvf s3sync.tar.gz

Copy s3config.yml.example to s3config.yml and edit it. Adding in your Amazon S3 keys. It’s also a good idea to setup the SSL certificate so your transfers to S3 will be https. Checkout the README.txt on how to do that. If you don’t care about that remove the -s flag from the script we will be writing in a minute.

(Optional: just to be on the safe side, you might want to make the s3config.yml file owned by the cron user that will run the backup, and then chmod 600 s3config.yml. It’s not a huge amount of security, but it’s better than nothing.)

Next up is the wrapper script that will run the s3sync ruby script. This is a simple script that just moves into the directory where we installed s3sync, and runs the script (I named it s3backup.sh):

#!/bin/sh

# This file syncs a direcotry with a specified
# bucket.
BACKUPDIR=/home/my_user_name/backups
SYNCDIR=/home/my_user_name/s3sync
S3BUCKET=server.robrohan.com
S3PREFIX=x
S3STORE=${S3BUCKET}:${S3PREFIX}

# move to the ruby sync direcotry where the .yml file is
cd ${SYNCDIR}
./s3sync.rb -r -s ${BACKUPDIR} ${S3STORE}

Obviously, you’ll want to change the paths to your install locations (or make them env variables or an include or whatever), the bucket name, and if you are not doing https, remove the -s flag.

Also, the S3PREFIX is any string you want it to be. It’s kind of like a drive letter inside the bucket - the “root” of the bucket so to speak. You can leave it as x.

At this point you should be able to run this script by hand (don’t forget chmod u+x):

$ ./s3backup.sh

And your backup directory should “sync” with S3. You can verify that by refreshing the view in cockpit (the gear on the right has “Refresh object listing”). You can also right click on an object to delete / download it.

I use the sync term is somewhat lightly because the ruby process doesn’t delete files that have been deleted from the backup directory. It will only add or update the files on S3 - hardly a show stopper though.

Cron’ing It

The last step is wrapping this all up into a cron job so you don’t have to think about it. The server should just back itself up.

To do this requires one more simple wrapper server_backup.sh:

#!/bin/sh

UTILS=/home/my_user_name/path_to_the_previous_scripts

# Create database backups
${UTILS}/dbbackup.sh database1 myusername1 password1
${UTILS}/dbbackup.sh database2 myusername2 password2

# Send the backups to S3
${UTILS}/s3backup.sh

Pretty straight forward. It creates backups of any databases we care about, and then calls the script to sync the backup directory with Amazon S3.

The last step is adding it to cron which you should probably already know how to do, but basically it’s:

$ crontab -e

and then adding a line something like:

0 2 * * 1,3,5 /home/my_user_name/path_to_the_previous_scripts/server_backup.sh

Which will do a backup at 2:00am on Monday, Wednesday, and Friday - tweak as needed of course.

Conclusion

When I first started coding almost no one out outside of Rome used source control (Rome meaning corporations). Now, if you’re not using source control on your projects people look at you like you’re nuts (and for good reason - it just makes sense to use source control). I think backups are following this same pattern. Everyone knows they should backup, but few actually do - and those that do probably don’t do it offsite or redundantly.

Amazon S3 (and similar services) offer pretty easy to use, low cost, redundant, high bandwidth, always on, offsite storage. If your data == your business, you’re nuts not to use Amazon S3 or something like it.

I hope this lifted some of the mystery of how to use this kind of service (the exercise did for me :D).

Interestingly, one of the main reason for Firefox over IE was “that it is based on the global internet standard - W3C, and that it works equally well under Microsoft, Mac or Linux.” Techtree.com reported. From what I could see during my testing of IE7, it did horrible on the Acid2 test (FireFox did quite a bit better, and Safari passed) so I could see why they made this choice if standards was their major criteria.

One of the stats in the article that struck me was the following:

Firefox reportedly has taken over almost 18 percent of the French internet browser market, and is soon approaching the European average of 20 percent.

Techtree.com

20%?! When did that happen?

It seems to me Europe and Japan are getting more and more technologically ahead of America - or are at least developing the newest stuff and coming up with the newest trends (see cell phones, ruby, tests scores, robotics etc). Who are you watching and catering to?

Are we supposed to call it FreedomFox now?

At the time I was in the midst of learning Objective-C, but I told him I’d check it out when I could. Well, this holiday break I checked it out. Here is a brief run down of my take of Ruby and Ruby on Rails and a bit of software for the Ruby Curious.

Ruby is object oriented, cross platform, and integrates well with other languages (as a scripting language should). The syntax is easy to pick up, and is a mix between Java/C++ and Perl.

Ruby comes built into Mac OS X 10.4 (just type ruby –version in a terminal to see), and can be used as a replacement, or enhancement to perl, shell scripts, or applescript. Having it built into OS X makes it easy to learn and tinker. You just write and run. Nothing to install…ahhhh… it’s good to be da’ king.

For a quick intro to Ruby try this. And if you don’t have a Mac and don’t want to install Ruby you can play with this online interactive shell.

Now, Ruby on Rails, which I am sure you’ve heard about by now, is a framework for writing web applications using Ruby as the scripting language. Much like Coldfusion+MachII, ASP+Something, (JSP+Beans)+Struts, etc, but with a strong focus on Model / View / Controller. It’s important to note that the two are separate - I like Ruby, but I don’t think I like Ruby on Rails (I’ve just started playing though).

I like the concept, but the thing about Ruby on Rails I don’t like very much is the fact that you just run a bunch of scripts to generate a bunch of code. I guess I am old because I hate it when code “just appears”. They say Ruby on Rails is great because you don’t need any config files, but what you get instead is a bunch of scripts that generate code that the noob has no idea where it goes or what it’s really doing.

Script kiddie web development? If it is, that worries me a bit.

I am all for saving time, and it could be that I just don’t fully understand it yet, but I’ve been burned by WSAD generated bean code too many times to just buy into this…

For a perfect example check out the “create a blog in 15 minutes” movie. He keeps saying “Notice I am not writing any code”, but tell me - what the hell are all those scripts he keeps running generating? Hopefully he knows what they are doing, and I hope the Rails developers do too.

In my opinion scripting is good to save time, but only *if you know what the scripts are saving you from*. Blindly running scripts to create code you don’t edit or understand seems like a recipe for disaster to me.

Slight rant aside, I’ve only been playing with Rails for a couple days and the jury is still out for me until I really understand what is going on. I do encourage you to check out Ruby and Ruby on Rails despite my vocal bias back there.

To try out Rails:
Instead of setting up a whole development server, setting up a database, and messing with Apache to get the whole Ruby on Rails environment going check out locomotive . It’s a full Ruby on Rails development server complete with a copy of Ruby, an integrated web server, and an sql server all on one dmg (Free). It wont mess with your system settings, Apache, or your local OS copy of Ruby. Plus, if you don’t like it you can just delete the app file - no harm no foul.

To edit Ruby (+Rails) code you can use Afae($), or Textmate($) which integrates with locomotive (and is the editor used in the above movies), or you can try a Ruby on Rails Eclipse plugin (Free), or BBEdit, or whatever.

In my opinion, there is something to the hype. Ruby is simple, fun, easy to learn, and powerful - I don’t know about Rails yet though.