I woke up Saturday with an idea (which is actually a joke at our house now, “Guess what honey, I have an idea!”). I wanted to have a podcast of news bits that I could listen to while I did my morning routine - the catch being I wanted it daily, and I wanted to control the shows overall subjects at will.

That desire, along with my new found Automator curiosity, led me to throw together Rohan Times Radio News ( :-D ). It’s an automatic process that takes an RSS feed, reads it, and makes a podcast out of the stories. To get the content I want, I am using yahoo pipes to filter and sort the feeds, and then republish the feeds as the shows teleprompt.

Here is a basic overview of how it works (sorry for the oddities with the movie, youtube seems to be doing odd things on upload lately):

One of the coolest parts about this is it basically requires no code (I did, however, use a bit of code to have finer control).

I don’t think I can publish the podcast for the world, as I think it would be seen as a copyright violation, but if you are interested in this let me know.

(I should preface this with using game emulators and and game ROMS is, sadly, of questionable legality)

Short and sweet: it seems XBox controllers (non 360) use USB to do their magic, so modifying an XBox controller to use with your computer is very simple. Just cut the XBox controller and the USB cords then wire them together (leaving out the yellow wire). The inside wire colors will match up so it shouldn’t be hard, but here is a movie showing how it’s done (ignore the driver part unless you are on windows).

Once your controller is ready, download the Mac OS X XBox HID driver. After it’s installed you can find the settings in System Preferences.

Finally, fire up Snes9x, and go into Config > Configure Controllers. You should see the following:

In this screen click on a button (for example the purple up arrow) then press the corresponding button on the controller (the direction pad up button, or the joystick up). Once the controller is configured you’re all set to play - and you didn’t even have to ruin your original SNES controllers.

It works by adding bones to vector drawings, and then you animate the drawings with the bones. It’s quite fun.

I’ve been learning how it works by making silly, pointless movies. I showed this one to my wife, and she was cracking up (mostly because the animation is so dorky I think).

Hopefully you’ll find it mildly entertaining (both my wife and I are quite mad, so our version of funny is not often other people’s).

The song is from ChinesePod.com, from the lesson Head, Shoulders, Knees, and Toes.

(for the astute Chinese learner they’ll notice the words are actually Hair, Shoulders, Knees, and Feet)

Recently, however, I’ve been getting into Git. Git is different from any source control system I’ve used. It’s a distributed source control system which not only gives it a huge work flow advantage in my opinion, but also means you can have repositories without a dedicated server.

The distributed nature

Every coder I’ve ever seen that is working on a project using source control has a work flow like the following:

1) Checkout code locally.
2) Work on code locally trying to add features / bug fixes.
3) If work spans several days then, “update” every day, and try to solve small conflicts with the main repository.
4) When done - commit, and if there are conflicts, resolve the conflicts.

The main, and slightly elusive, point I’ve noticed with this setup is the repository is nothing but a database. All the work, all the code managing is done by the developer and then “pushed” to the repository database when ready.

The developers local copy is really just a branch of the main repository. So why not let the developers local copy have all the rights and privileges to be someone else’s main repository? Git was built with this idea in mind, and the explicit goal of being able to merge all changes back upstream.

What this means in reality, is branching and merging are at the core of git, and I’ve found it to work really well. It lets you test wild ideas with code, and not worry about how you’re going to rollback your changes if you go to far out, and it doesn’t work - if that makes any sense.

Look ma, no server

The other exciting part about git, for me, is you don’t need a server running a process anywhere - you don’t need a git server running somewhere.

Most places I’ve worked I’ve implemented source control, and managing a source control server is time consuming. This is all subjective to the level of funding, number of people on staff, tools available, amount of branching etc, but what if you could just remove the overhead of another service that someone has to manage?

By the way, quick plug, I often use cvsdude.com for my Subversion outsourcing, but even with the tools they offer someone still has to manage the Subversion server (adding users, permissions, the cost).

While I feel you’re insane if you don’t use source control, sometimes running an entire Subversion server is overkill for a project. Writing a Mac OS X widget for example - do you need a full on server, bugzilla, a wiki, and trac hooks for that?

With git, however, you can just put a copy of the repository on a server, and call it your main repository. Done. You can control push access using SSH logins, and if you want to allow anonymous access to the repository, you can make it available over HTTP.

Here is what this looks like, for my notes and if you want to try it yourself:

Get a git repository. Either a new one:

$ mkdir my_cool_project; cd my_cool_project;
$ git-init
$ cp -R [all_my_code_files] ./
$ git add .
$ git commit -a

or copy a whole subversion repository (comments changes and all):

$ git-svn clone -T trunk https://mystuff.com/my_project

Do your stuff

$ write_lots_of_amazing_code.sh
$ git commit -a

Prepare for publish:

$ pwd
/home/guy/my_cool_project
$ git repack -d
$ git clone --bare -l . /tmp/your-proj.git
$ git --bare --git-dir=/tmp/your-proj.git update-server-info
$ mv /tmp/your-proj.git /home/guy/Desktop

Now upload the your-proj.git folder to your publicly available server. As I said before, if you want to allow anonymous access put it in a HTTP available folder, if you want a private one, put it in your home directory or something.

Turn on the hooks for it to update itself when you “push” your new code changes. (one thing to note here is the server needs to have git installed - sudo apt-get install git git-core on Debian / Ubuntu systems)

$ uname -a
your_server GNU/Linux
$ pwd
/var/www/myrepo.com/your-proj.git
$ git --bare update-server-info
$ chmod a+x hooks/post-update

Now it’s ready to be used as a central server.

To “checkout” the code - really cloning the repository locally:

$ uname -a
your_development_box
$ git clone ssh://[username]@[server]/var/www/myrepo.com/your-proj.git

Then working locally, you commit your changes locally.

$ git commit -a

It is very easy to make branches and merge them back into your local trunk (often named master). Here is an in depth posting on the subject when you are ready.

To update your local repository to the code on the server:

$ git pull

And when you are all set with your local changes and ready to push them to the server:

$ git push

Anonymous checkout:

$ git clone http://myrepo.com/your-proj.git

Last bits

Some things to consider:

* Git doesn’t work on windows without Cygwin. It works great on Mac OS X and Linux however.

* There are very few GUI tools for git, you (and your developers) need to be cool with the terminal. (try gitk from the command line)

* You will have many full copies of your repository floating around. While this is great for backup purposes, it might be bad for some projects.

* On Mac OS X I think you have to compile Git from source (I did anyway). It’s not hard, but if that scares you…

I guess I should start out with what Amazon S3 is I suppose… It’s a service that you can use to store reams of data, offsite, online, accessible anytime for a decent price. As of this writing the costs are:

Storage: $0.15 per GB-Month of storage used
Data Transfer: $0.10 per GB - all data transfer in

$0.18 per GB - first 10 TB / month data transfer out
$0.16 per GB - next 40 TB / month data transfer out
$0.13 per GB - data transfer out / month over 50 TB

see Amazon S3 for more details. If you use .mac, you can think of it like an iDisk on steroids - or a huge webdav server - but it’s not just one sever it’s a farm of servers with huge amounts of redundancy.

My goal here is to make a simple process that will automatically back up some of my web sites - for this example I am using this blog. What this process is going to do is backup the blogs MySQL database. This will just be a basic setup, but you can tweak it or add to it as you see fit.

The Plan

We are going to make some scripts that dump the database to a directory, and then sync that directory with the Amazon Service. The sync will work somewhat like rsync so if we want to add other files to the sync tree, it will just be a matter of adding them into sync area.

Amazon S3 Account

To start off, you’ll need an S3 account. You can sign up (and login) here. As you can tell, I like the Amazon Storage Service; however, their web site lacks in the usability department in my opinion. So if you get a bit confused trying to navigate the site, don’t worry you’re not alone.

After you go though the account setup process, you’ll need to get your access key id (like your username) and your secret access key (like your password). To find these hover over the Your Web Services Button button on the top right, and click AWS Access Identifiers (See shot below).

From that screen, on the right you should see your Access Key ID, and be able to show your Secret key. You’ll need these for the next steps.

Using S3 From the Desktop

You’ll likely want access to your S3 account from your desktop. If only to verify the backups are running. However, once you get this setup, see how well it works, and see how inexpensive it is you’ll probably start using it for other stuff (as I do :D).

For desktop access there are quite a few choices. I went the cheap route. I use cockpit. It’s a Java based application that lets you use your S3 account from the desktop. It has the feel of an FTP application. Jungle Disk is another one I hear is very good. It mounts the S3 account as if it’s a remote drive (on Mac) or a drive letter on windows (and it also works on Linux) - Jungle Disk is $20 USD.

I am going to use cockpit in this article, but they should all work about the same.

Start up cockpit. On Mac / Linux, run the file:

$ /Applications/jets3t-0.5.0/bin/cockpit.sh

(on windows it’s the cockpit.bat file)

When it first starts up there will be several ways to login. You can read up on the choices on the cockpit website, but for now we’ll use the Direct Login tab. On this tab is where you’ll put the keys you got from the S3 Account page (see image)

S3 uses things called buckets. You’ll see that term quite a bit when reading about S3. It’s just a place to store stuff. You can think of it as a drive letter (if you’re coming from a windows world), or a database table, or whatever. What we need to do here is make a bucket for our backups. I made one called server.robrohan.com - you can name yours anything you like. The shot below is of my server.robrohan.com bucket and a few backups that have already run.

To create a new bucket, click the gear icon on the left and select Create new bucket…. You’ll need at least one bucket to continue with this article.

Database Dump

Now that we have an Amazon S3 account setup, we can access it from our desktop, and we have one bucket ready to hold stuff we can start doing our backups. The first step in doing a backup of our MySQL database is to get a dump of the database. If you already have a process for this you can skip this section. (Note: This is a very simple way to get a backup of a MySQL database there are better, more efficient ways to do this.)

Here is a simple script to dump a database to a bziped file. You an call this script dbbackup.sh, and it is written so it can be called from cron. (This is pretty insecure, so be sure you’re ok with that or seek help in making it more secure if you need)

#!/bin/sh

# This file backs up a database to a backup dir
# that will be synced with s3

if [ $# != "3" ]; then
    echo “Usage:”
    echo ”      $0 [database] [username] [password]”
    exit 1
fi

DATE=`date +%d`
DB=$1
USER=$2
PASS=$3

BACKUPDIR=/home/my_user_name/backups
DBSTORE=${BACKUPDIR}/database

# create database backup
mysqldump –add-drop-table \
          -h localhost \
          –user=${USER} \
          –password=${PASS} \
          ${DB} \
          | bzip2 -c > ${DBSTORE}/${DB}_${DATE}.sql.bz2

Note the variable BACKUPDIR. This should be a directory you create somewhere on your server that you want to “sync” with the Amazon service. Meaning that anything you put in that directory will get backed up to S3. In this article we are just putting a database backup in the tree, but you could put server code, documents on the server, copies of configuration files, or whatever. The main point being anything in that directory will get copied to S3 (recursively - subdirectories as well).

The script is run like the following:

$ ./dbbackup.sh mydatabase myusername mypassword

And would produce the file:

/home/my_user_name/backups/database/mydatabase_[day_of_month].sql.bz2

In this process database backups will only go back a month. The first of the month will overwrite the last month’s first of the month backup. If you’d like to archive forever, just change the DATE variable to be `date +%y%m%d`. For my purposes, doing backups everyday is overkill.

Syncing with S3

So now the meat of the thing. There are, as in the desktop access, many server side scripts you can use to write data to S3. What this example is going to use is s3sync.rb. s3sync is a ruby script that acts much like rsync. There is also an outdated perl version, and even a bash version. You can try others if you’d like, but for this I’ll be using the ruby version. That, of course, means you’ll need ruby on the sever (not rails or anything, just ruby).

(If you don’t have ruby you can do apt-get install ruby, yum install ruby, ports install ruby, or whatever. It’s often very easy to get ruby on your *nix box)

Download and untar the s3sync file:

$ wget http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz
$ tar -zxvf s3sync.tar.gz

Copy s3config.yml.example to s3config.yml and edit it. Adding in your Amazon S3 keys. It’s also a good idea to setup the SSL certificate so your transfers to S3 will be https. Checkout the README.txt on how to do that. If you don’t care about that remove the -s flag from the script we will be writing in a minute.

(Optional: just to be on the safe side, you might want to make the s3config.yml file owned by the cron user that will run the backup, and then chmod 600 s3config.yml. It’s not a huge amount of security, but it’s better than nothing.)

Next up is the wrapper script that will run the s3sync ruby script. This is a simple script that just moves into the directory where we installed s3sync, and runs the script (I named it s3backup.sh):

#!/bin/sh

# This file syncs a direcotry with a specified
# bucket.
BACKUPDIR=/home/my_user_name/backups
SYNCDIR=/home/my_user_name/s3sync
S3BUCKET=server.robrohan.com
S3PREFIX=x
S3STORE=${S3BUCKET}:${S3PREFIX}

# move to the ruby sync direcotry where the .yml file is
cd ${SYNCDIR}
./s3sync.rb -r -s ${BACKUPDIR} ${S3STORE}

Obviously, you’ll want to change the paths to your install locations (or make them env variables or an include or whatever), the bucket name, and if you are not doing https, remove the -s flag.

Also, the S3PREFIX is any string you want it to be. It’s kind of like a drive letter inside the bucket - the “root” of the bucket so to speak. You can leave it as x.

At this point you should be able to run this script by hand (don’t forget chmod u+x):

$ ./s3backup.sh

And your backup directory should “sync” with S3. You can verify that by refreshing the view in cockpit (the gear on the right has “Refresh object listing”). You can also right click on an object to delete / download it.

I use the sync term is somewhat lightly because the ruby process doesn’t delete files that have been deleted from the backup directory. It will only add or update the files on S3 - hardly a show stopper though.

Cron’ing It

The last step is wrapping this all up into a cron job so you don’t have to think about it. The server should just back itself up.

To do this requires one more simple wrapper server_backup.sh:

#!/bin/sh

UTILS=/home/my_user_name/path_to_the_previous_scripts

# Create database backups
${UTILS}/dbbackup.sh database1 myusername1 password1
${UTILS}/dbbackup.sh database2 myusername2 password2

# Send the backups to S3
${UTILS}/s3backup.sh

Pretty straight forward. It creates backups of any databases we care about, and then calls the script to sync the backup directory with Amazon S3.

The last step is adding it to cron which you should probably already know how to do, but basically it’s:

$ crontab -e

and then adding a line something like:

0 2 * * 1,3,5 /home/my_user_name/path_to_the_previous_scripts/server_backup.sh

Which will do a backup at 2:00am on Monday, Wednesday, and Friday - tweak as needed of course.

Conclusion

When I first started coding almost no one out outside of Rome used source control (Rome meaning corporations). Now, if you’re not using source control on your projects people look at you like you’re nuts (and for good reason - it just makes sense to use source control). I think backups are following this same pattern. Everyone knows they should backup, but few actually do - and those that do probably don’t do it offsite or redundantly.

Amazon S3 (and similar services) offer pretty easy to use, low cost, redundant, high bandwidth, always on, offsite storage. If your data == your business, you’re nuts not to use Amazon S3 or something like it.

I hope this lifted some of the mystery of how to use this kind of service (the exercise did for me :D).

TextEdit can save HTML with images in a file format called WebArchive, but in order to put the files on a web server for the world to view, the files need to be extracted from that format into a normal directory structure.

I found a utility on Sourceforge that would extract the file from a WebArchive format, but it seemed to mangle the src attribute of the image tags. Since the application was so close to what I wanted, I decided to tweak the code to my liking.

However, the source code provided on Sourceforge was incomplete - it was missing the .nib files. So I decided to just fork that project and put together my own Web Archive Extractor utility using most of that project’s code as a base.

After I got it all setup, it seems the core worked correctly without me changing anything. Meaning the image references where getting generated correctly. So either the binary distribution is not the same as the source, or I just messed up somehow when originally extracting.

Either way, my version works like I want it to, and it looks better ^_^

If you’re looking for such a thing, you can download it on the project page.

(Safari can also save whole web pages in WebArchive format - images, css, js, and all - by selecting Save As… If you are leaning something about a site, you can do that and then look over the site’s code at your leisure.)

There are a few ways to get around the AdBlock Firefox plugin, and I’ve yet to see any site do it. For a hoot, here is one way to get around Adblock. (My teaser post a year ago gets a heck of a lot of traffic, and no one offered me 5 million for this - doh)

(If this post makes you upset: While this isn’t really an exploit, it can kind of be seen that way. Isn’t publishing exploits good for security? This should just make AdBlock stronger in the end.)

One flaw of Adblock is that you have to deny images, or explicitly block ads. Adblock makes this easy by letting you do regular expressions to block ads, but this also means that you have to find patterns in the ad URL. So the simplest way around Adblock is to never create ad URLs that have a pattern.

This will become an arms race at some point, but as long as you can keep URLs random - file name, directory name, and directory levels - it should be near impossible to create a regular expression to stop the ads. In the example, the URL is only mildly random, and using a real word every once in a while should make it even more difficult to defend against (though I think even the example would be hard to keep ads from showing up).

One other thing required for this technique to work is the ads have to be served from the same domain as the other content - or at least appear that way. If you use an ad server that supports RPC calls to get the ads to display, or if you serve your own ads you can use this technique.

The reason the ads need to come from the same domain is because it makes creating a rule like http://robrohan.com/* a bad thing. Doing a rule like that will stop the ads, but it will also stop every image on the site. If you have at least somewhat interesting content (say an image in the article), they are less likely to create that rule.

So here is how you use the technique. (This stuff is not really new, just applied in an effort to subvert adblock)

The example uses Apache mod_rewrite to force URLs that look a certain way to the ad generation page.

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{QUERY_STRING} ^(.*)$
RewriteRule ^([a-zA-Z0-9\/]+)/([a-zA-Z0-9]+)(.*)$ ad.php
</IfModule>

You can tweak this to your liking / needs, but, again, you’ll want the randomness to start at the root of the site. Most frameworks hit the index page on the root and delegate work out from there. If you are using a framework like that you should be golden, else it should still work with a bit of tweaking. In the example, it calls another php page with no problems for example.

You could do the mod rewrite stuff with straight php, asp, cf, ruby or whatever too if you want (or if you use IIS).

Once the when-you-get-some-random-stuff-send-the-request-to-the-ad-generation-page process is setup, you can write some sort of random url generation function that will cause that rule to fire. Then use this function everywhere you want to show an ad. In the example I am using:

function ad_url($base) {
	$levels = rand(1,10);
	$fake_dir  = '';

	for( $x=0; $x<$levels; $x++ ) {
		$bit_o_stuff = md5(uniqid());

		$part = substr($bit_o_stuff, rand(1, strlen($bit_o_stuff)), rand(1,strlen($bit_o_stuff)) );
		$fake_dir .= $part . '/';
	}

	return $base . $fake_dir;
}

Which basically just creates a path that could be between 1 and 10 levels deep with random directory names. The urls it spits out would look something like:

http://robrohan.com/UnAdblock/4e9800998ecf8427e/0b204e9800998ecf8427e/800998ecf8427e/8/98ecf8427/8f00b204e9800998ecf8427e/0998ecf/f8427e/204e9800998/204e9800998ecf8427e/
or
http://robrohan.com/UnAdblock/27e/04e9800998ecf8427e/
or
http://robrohan.com/UnAdblock/b20/8ecf8427e/9//998ecf8427e/427e/e/98f00b204e9800998ecf8427e/427e/

(remember for the example it’s in a subdirectory, but you’d want the URL from the root when doing it for real)

The last part is the ad.php file which just loads and shows some ad:

header('Content-Type: image/jpeg');
//lots of hardcoded stuff, but you get the idea
$x = 0;
$images = array();

//get a listing of files from a directory or from a database
for ( $x=0; $x<=3; $x++) {
	$images[] = $x . '.jpg';
}

$full_path_start = str_replace('ad.php', '', $_SERVER['SCRIPT_FILENAME']);
$dsp_image = rand(0,3);
echo file_get_contents($full_path_start . 'images/' . $images[$dsp_image]);

The result can be seen on the example page.

Using this in the real world you’d have to have a way to know which ad you were showing to create the proper anchor tag - you could have the first number of each subdirectory be a digit of the ID or something, but remember patterns are your enemy.

Anyway, I hope that was fun. And I eagerly look forward to being bombarded with ads.

The ride is about 6 miles one way. Depending on the head wind, and how I feel about taking on the hills that day, the trip takes 35-45 minutes.

In my Chinese class, I am memorizing poems to work on my tones. It seemed like there should be a way to use that extra hour plus of pedal time to study without killing myself.

Here is my simple, cheap way of having my reference material available while I ride. (This is a really, really simple hack).

By the way, this is not something you can use to read while riding (unless you are crazy). This will probably only be useful if you are trying to memorize something, and doing a quick glance down is enough to check yourself.

This is what it looks like when you’re done:

You can probably figure out how to do it from the picture, but here is how I put mine together.

Get a sandwich bag and three twisty ties. The ties can be anything really, but I happen to have some fancy cable ties.

Cut 4 small slits on the back of the sandwich bag. Two horizontally below the zipper, towards the top, on the left and right side. Then cut two slits vertically towards the middle of the bag about an inch apart.

Then feed the ties though the top slits, and secure the ties to the handle bars. Finally, feed the other tie though the middle slits, and secure it to the bottom of the handle bars.

You can fold a piece of paper with a good amount of it showing, and slip it in the sandwich bag. If you do it right, you can close the sandwich bag to keep water out.

You could also use it to hold a map, but it probably wont stand up to serious mountain biking. However, I’ve been using the same bag for two weeks now and it’s still going strong.