n8 foo

Ruby on Rails dev environment on a remote server

noreply@blogger.com (Nathan) — Fri, 25 Dec 2009 00:54:00 +0000

A friend asks:

...so I'm trying to install RoR on a remote server of mine, but it seems that RoR development methodology wants you to test your webpages on localhost:3000. But, obviously, since the computer is in the cloud, I can't just well pull up a webpage on the console. Is there an easy way to get development pages to show up on a web browser that's not local?

Ruby on Rails was designed by a bunch of guys writing code locally on their Macs so it's easy to see why they assume a local web sever is a solution. So there are a couple of options.

1) use mod_passenger w/Apache for all your environments. This requires quite a bit more work but it's the way you'd want to do it for any larger environment. Google will teach you all you want about this, but I recommend it only when your environments are mature and you are needing to scale out production.

2) Tell Apache to pass dev.yourdomain.com to localhost port 3000. You'd need to set up a dev hostname in DNS and point it at the server as well. It would look like this in the apache config:

<VirtualHost *:80>
       ServerAdmin         webmaster@domain.com
       ServerName          dev.domain.com
       ProxyPreserveHost   On
       ProxyPass           /      http://127.0.0.1:3000/
       ProxyPassReverse    /      http://127.0.0.1:3000/
</VirtualHost>

3) Similar as above, but if you don't want to mess with DNS, you can
proxy based on the URL:

ProxyPass /foo http://127.0.0.1:3000
ProxyPassReverse /foo http://127.0.0.1:3000

more on mod_proxy here: http://httpd.apache.org/docs/2.0/mod/mod_proxy.html

4) Use ssh to tunnel requests from your local machine out to the server (ssh tunnel). ssh will forward the tcp packets across the pipe towards a host you specify on the other end. In this case, you'd tell it to send localhost port 3000 (on your laptop) across the pipe and out to localhost port 3000 on the server. This will only provide access for you (since it's a private tunnel).

# ssh -L 3000:127.0.0.1:3000 host.server.com

Then in your browser surf to http://127.0.0.1:3000

For speed, I'd probably do #4 to test that things are working and go up towards #1 as you get your environment more set up. #1 is the most amount of work (mod_passenger can be a PITA).

wildcard subdomain SSL certs

noreply@blogger.com (Nathan) — Tue, 22 Dec 2009 09:10:00 +0000

A friend asked:

If I bought a wildcard certificate for *.domain.com, wouldn't that cover
*.sub.domain.com?

Hrm...I had to look that one up. The answer is: no, not accorindg to the RFC. RFC 2818 states:

Matching is performed using the matching rules specified by
[RFC2459]. If more than one identity of a given type is present in
the certificate (e.g., more than one dNSName name, a match in any one
of the set is considered acceptable.) Names may contain the wildcard
character * which is considered to match any single domain name
component or component fragment. E.g., *.a.com matches foo.a.com but
not bar.foo.a.com. f*.com matches foo.com but not bar.com.

more here: http://www.ietf.org/rfc/rfc2818.txt

There are reports that older versions of Firefox don't complain when encountering an out of spec sub-domain SSL wild-card but IE would. I would recommending sticking with the RFC spec.

System Administration with Capistrano

noreply@blogger.com (Nathan) — Wed, 25 Nov 2009 04:24:00 +0000

In previous years, when I had to run commands on hundreds or thousands of servers, I'd hack together a combo of expect and perl. It would almost always evolve into a system of complicated config and command files that was rickety and didn't handle errors well. I didn't dare mess with multi-threaded perl, which meant it was serial execution and slow for large clusters. It got the job done but left me wishing for a better system. I have always had cfengine in my sysadmin toolbox, but it's more about entropy reduction and not set up for one-off or occasional situations. I tried a few parallel shell implementations (such as dsh, pdsh) and found them all lacking.

Enter Capistrano. It bills its self as an 'easy deployment' system, with Ruby on Rails application deployment as the main use case. And since I'd never worked in a RoR environment before, I had no real reason to look into it much. But in the last 3 months, I have worked at 2 different companies that use RoR + Capistrano for deployment and have learned enough to see it's true power. How I'd describe it to a fellow sysadmin is: "parallel execution of scripts and commands on multiple hosts...easily". Want to quickly execute a command on every host in your cluster? This is the way to do it.

Installing it is pretty easy...you need a modern version of Ruby, a modern version of RubyGems and then a 'gem install capistrano' later, and you're good to go. You only need to install all of this on the controlling/deployment server....not on all your clusters/nodes. If you get errors with the version of ruby/gems that comes with your distro, install from source (recommended). I followed this tutorial to get it set up, and to get the basics. You sould read it as well. They skip a few necessary things (such as sudo and useful ENV variables) which I have below.

An example Capfile of how to restart apache on a whole cluster:

role :apache_cluster,
"www1",
"www2",
"www3"

desc "restart apache on www hosts"
task "restart_apache_www", :roles => :apache_cluster do
sudo "/etc/init.d/apache2 restart"
end

'sudo' is a built in method of modern versions of Capistrano. Instead of the 'run' method, you use 'sudo' and it understand and responds to the prompt (if prompted). Very slick. One thing to keep in mind is that is is running everything as YOU, unless otherwise specified. It will look like you logged into 50 servers all at once and ran sudo commands all at once. I bet that'd look cool on a Splunk log graph.

Now, from the command line, type 'cap -T' to get a list of your documented commands. As long as you describe your commands, you will always get a list of what you can run. 'cap -e command' will explain commands.

$ cap -T
cap invoke              # Invoke a single command on the remote servers.
cap restart_apache_www  # restart apache on www hosts
cap shell               # Begin an interactive Capistrano session.

Run the command we set up: 'cap restart_apache_www'. It will prompt for your password.

$ cap restart_apache_util
* executing `restart_apache_www'
* executing "sudo -p 'sudo password: ' /etc/init.d/apache2 restart"
  servers: ["www01.domain.com", "www02.domain.com", "www03.domain.com"]
Password:
[www01.domain.com] executing command
[www03.domain.com] executing command
[www02.domain.com] executing command
** [out :: www01.domain.com] * Restarting web server apache2
** [out :: www01.domain.com] ...done.
** [out :: www03.domain.com] * Restarting web server apache2
** [out :: www03.domain.com] ...done.
** [out :: www02.domain.com] * Restarting web server apache2
** [out :: www02.domain.com] ...done.
command finished

And that was completed in parallel, in about 1 second. What if you have a one-off thing you want to run on all hosts? Try 'cap invoke', no Capfile required. If you have a Capfile with hosts defined, it will run against all of them by default, or it can take a role by passing ROLE as an env variable.

$ cap COMMAND=uptime HOSTS="www1,www2" invoke
* executing `invoke'
* executing "uptime"
    servers: ["www1", "www2"]
Password:
[www1.prod] executing command
[www2.prod] executing command
** [out :: www1] 16:57:04 up 190 days,  4:30,  0 users,  load average: 0.30, 0.33, 0.33
** [out :: www2] 16:57:04 up 190 days,  4:42,  0 users,  load average: 0.42, 0.32, 0.32
command finished

and

cap ROLES=www COMMAND=uptime invoke
* executing `invoke'
* executing "uptime"
    servers: ["www1", "www2", "www3"]
Password:
[www1] executing command
[www2] executing command
[www3] executing command
** [out :: www1] 17:00:17 up 190 days,  4:33,  0 users,  load average: 0.54, 0.37, 0.34
** [out :: www2] 17:00:17 up 190 days,  4:46,  0 users,  load average: 0.18, 0.27, 0.29
** [out :: www3] 17:00:17 up 190 days,  5:02,  0 users,  load average: 0.17, 0.22, 0.25
command finished

But every time you 'invoke', you must re-type your password. Want to stay connected? Try the shell:

$ cap shell HOSTS="www1,www2"
* executing `shell'
====================================================================
Welcome to the interactive Capistrano shell! This is an experimental
feature, and is liable to change in future releases. Type 'help' for
a summary of how to use the shell.
--------------------------------------------------------------------
cap> uptime
[establishing connection(s) to www1, www2]
Password:
** [out :: www1] 17:03:24 up 190 days,  4:36,  0 users,  load average: 0.29, 0.32, 0.32
** [out :: www2] 17:03:24 up 190 days,  4:49,  0 users,  load average: 0.35, 0.30, 0.29
cap> w
** [out :: www1] 17:03:37 up 190 days,  4:36,  0 users,  load average: 0.24, 0.31, 0.31
** [out :: www1] USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
** [out :: www2] 17:03:37 up 190 days,  4:49,  0 users,  load average: 0.30, 0.29, 0.28
** [out :: www2] USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
cap> ls /tmp/blah
*** [err :: www1] ls: cannot access /tmp/blah
*** [err :: www2] ls: cannot access /tmp/blah
*** [err :: www1] : No such file or directory
*** [err :: www2] : No such file or directory
error: failed: "sh -c 'ls /tmp/blah'" on www1,www2

Notice that errors show up with a 'err' line.

While useful and timesaving, this barely scratches the surface of the power of Capistrano. I suggest you read the "From the Beginning" doc on the Capistrano site. If you discover any cool recipes, share them in the comments on this blog and I'll publish them as a followup later (as I learn more recipes myself).

P.S. I think I'm going to add a feature to MachDB to export host lists in a format compatible with the Capfiles.

DIY Encrypted Password Vault

noreply@blogger.com (Nathan) — Mon, 30 Mar 2009 17:33:00 +0000

This is something I've needed at various jobs/situations for years...a place to store the root/router/database/web passwords that only I can see. There are a lot of desktop/handheld apps for this but I always feel like I could lose the computer/handheld that it's on and I'd be boned. I'd rather have something I can stick on a server somewhere and access via a remote shell....or carry it around on a thumb drive. Here are the scripts:

encrypt.sh

#! /bin/sh 

openssl bf -a -salt -in $1.txt -out $1.bf && rm -v $1.txt

decrypt.sh

#! /bin/sh 

openssl bf -a -d -salt -in $1.bf

To use it, create a file named blah.txt that has your secret info in it. Run the encrypt script first:

$ ./encrypt.sh blah
enter bf-cbc encryption password:
Verifying - enter bf-cbc encryption password:
removed `blah.txt'

It will encrypt the file and remove it. Check the contents of the file:

$ cat blah.bf
U2FsdGVkX1/+ZGiXPSZX8MED9aXrm1NfIEjpv5vvFKo=

It's actually base 64 encoded so you can email it to yourself for safe keeping if you so choose.

To decrypt for reading:

$ ./decrypt.sh blah
enter bf-cbc decryption password:
secret host: secret password
secret host2: secret password2

Now take the encrypted output file and the 2 scripts, email it to yourself and store a copy on a thumb drive. :)

Why all phones need a silent ring

noreply@blogger.com (Nathan) — Mon, 23 Mar 2009 16:28:00 +0000

Telemarketers, vendors and people I'd rather not communicate with frequently intrude on my early morning slumber (esp East Coast vendors), meetings, lunches, free time and life in general. And since they usually call from unrecognized numbers, I feel compelled to answer (could be something important, right?) A co-worker and I have been using a neat technique to remove these individuals ability to communicate with us...create a new contact called "Do Not Answer" with a custom silent ring tone. Each time they call from a new number, add them as an additional number to that contact. And with that silent ring, now they can't interrupt you in meetings, at home, early in the morning, etc.

I used iTunes to make a silent ringtone...you can download it here: iPhone Silent Ringone

automated nmap scans

noreply@blogger.com (Nathan) — Fri, 13 Mar 2009 23:14:00 +0000

Whipped this up for work, figured I'd share with the world, since it's decently useful. Stick it in cron nightly, needs to run as root. It will run a diff on what it sees and email you if there are new ports/hosts that pop up on your networks. If you find errors or mods, use this: http://pastebin.com/f635a7517 to modify it and post in the comments.

#! /bin/sh

DIR="/opt/nmap/scans"
NETWORKS="192.168.1.0-255"
TODAY=`date +%Y%m%d`
YESTERDAY=`date -d yesterday +%Y%m%d`


for network in $NETWORKS
do
      nmap -n -sS $network -oG $DIR/$network.$TODAY.nmap
done


for network in $NETWORKS
do
      diff -I "^#" $DIR/$network.$TODAY.nmap $DIR/$network.$YESTERDAY.nmap  > $DIR/$network.$TODAY.diff
done

for network in $NETWORKS
do
      SIZE=`find $DIR/$network.$TODAY.diff -size +0b`
      if [ "$SIZE" = "$DIR/$network.$TODAY.diff" ]
      then
              cat $DIR/$network.$TODAY.diff | mail -s "Change Detected for $network"  user@host.com
      fi
done

automated disk partitioning with sfdisk

noreply@blogger.com (Nathan) — Sun, 04 Jan 2009 04:20:00 +0000

I discovered sfdisk a few years ago (part of util-linux) and have been using it in automation scripts ever since. sfdisk is like fdisk, but is scriptable. So for example, to list the partitions on a disk:

[root@host]# sfdisk -l /dev/sdc

Disk /dev/sdc: 121601 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdc1          0+ 121600  121601- 976760001   83  Linux
/dev/sdc2          0       -       0          0    0  Empty
/dev/sdc3          0       -       0          0    0  Empty
/dev/sdc4          0       -       0          0    0  Empty

To list them in a dump format, suitable as input to sfdisk (for cloning, saving or for some wacky awesome script):

[root@host ]# sfdisk -d /dev/sdc
# partition table of /dev/sdc
unit: sectors

/dev/sdc1 : start=       63, size=1953520002, Id=83
/dev/sdc2 : start=        0, size=        0, Id= 0
/dev/sdc3 : start=        0, size=        0, Id= 0
/dev/sdc4 : start=        0, size=        0, Id= 0

You can use that dump in a fashion like this to clone a disks's partition map:

sfdisk -d /dev/sdc | sfdisk /dev/sdd

Or for saving it and using it later:

sfdisk -d /dev/sdc > partition.sfdisk
sfdisk /dev/sdc

End of 2008

noreply@blogger.com (Nathan) — Thu, 01 Jan 2009 07:44:00 +0000

I have done a few things in the last few months that are worthy of mention. I haven't had much of a chance to blog about them or write them down, what with them all being back to back and then holidays, being sick, vacation, more holidays, more being sick. But here are some links to the media I've produced. Enjoy.

San Diego CA USA to Cabo San Lucas, Baja California Sur, MX (in 6 days, 2200+ miles, on my new BMW adventure motorcycle)

One month of beard growth in 5 seconds (an experiment in time lapse)

Deleting that same beard at high speed (an experiment with tracy's camera)

The mechanical wonderment behind a bowling alley machine

ssh tab completion on known_hosts

noreply@blogger.com (Nathan) — Wed, 05 Nov 2008 22:35:00 +0000

It's silly I've waited this may years to go figure this out. Many of you may already know that modern installs of OpenSSH will tab complete hostnames based on what's in the /etc/hosts file. But there is a neat little addition to your .bashrc that will tack on the ability to tab complete hostnames based on what's in ~/.ssh/known_hosts. Add this to your .bashrc:

SSH_COMPLETE=( $(cat ~/.ssh/known_hosts | \
cut -f 1 -d " " | \
sed -e s/,.*//g | \
uniq ) )
complete -o default -W "${SSH_COMPLETE[*]}" ssh

All your new shells will auto complete based on what hosts you've connected to once (and therefore have entries in the known_hosts file). Any host you've never visited, well it won't be there. If you want to filter it based on certain hosts (for example, hosts in a certain domain name), just add a | grep domain.com after the uniq. If you're like me, this will save a lot of keystrokes over the next few years.

Tip: If you cut and paste my text above and it gives errors, make sure your cut-n-paste didn't change the quotes. If you want to see what it's going to use (or troubleshoot/modify), you can run this on the command line:

cat ~/.ssh/known_hosts | \
cut -f 1 -d " " | \
sed -e s/,.*//g | \
uniq

New Midpipe

noreply@blogger.com (Nathan) — Sun, 28 Sep 2008 03:26:00 +0000

Out for a hyperspin
Originally uploaded by n8foo

Installed a new LeoVince midpipe on the Hypermotard earlier in the week (2am in the garage with a rubber hammer the night before an early meeting). Finally got a moment away from the keyboard to take it out for a spin tonight. Total awesomeness. It starts better, sounds better and runs better. I think it's a bit faster too. How awesome is that for the effort and price? I should have done this a long time ago.

Pics: Old and Busted. New Hotness.

I had also adjusted the bars a bit to be a tad higher, but I didn't like it. The turn signals are pointed at the ground 5 feet in front of me, which means my effective road use brightness went from 'are those lasers?' to 'dead lightning bug'. I may pick a position half way between this position and neutral. Will have to test it again later in the week when I've got some more time.

HD Video Test

noreply@blogger.com (Nathan) — Wed, 13 Aug 2008 06:32:00 +0000

Finally got myself a video camera. Been looking at them critically for about 8 years now...ever since my Hi8 cam broke in 2000. I loved making movies back then, and even lately have been doing various time-lapse type videos...not really 'movies' since they were shot with a SLR. A few years ago I got interested in video again with the advent of consumer level HD cameras. Canon came out with the HF10 and HF100, soda-can sized cameras that record 1080p HD to SDHC flash cards...no moving parts, great for strapping to cars/motorcycles, etc. When the price finally fell to a range I could deal with, I pulled the trigger. It's so cool...and little! This is my first test. Video quality is great! I hope to be posting more mischief and fun events in the coming months.

Is Vimeo the best place to host HD video? Anyone? Bueller? Anyway, enjoy some test shots from the backyard...

HD Test - Backyard Oddities from nathan hubbard on Vimeo.

changing your server over to GMT

noreply@blogger.com (Nathan) — Tue, 05 Aug 2008 06:08:00 +0000

Funny how it's kind of hard to find documentation on this fairly simple task. The host I'm working with is CentOS 5, a variant of Redhat Enterprise 5. I use the following commands:

cat /usr/share/zoneinfo/GMT > /etc/localtime

That is actually what sets the time. It's a binary file and is what our system tools and libraries look at.

vi /etc/sysconfig/clock

Edit the ZONE to be GMT. Mine looks like this:

ZONE="GMT"
UTC=true
ARC=false

Verify it looks right with these commands.

date ; date -u ; /sbin/hwclock --show

Reboot, done!

Statistics, Charts and R

noreply@blogger.com (Nathan) — Mon, 02 Jun 2008 04:51:00 +0000

For a few weeks now, I've been playing around with the statistical computing package "R" to draw some graphs/plots of data. Why would I want to do this? Let me quote one of my favorite movies, Sneakers:

The world isn't run by weapons anymore, or energy, or money. It's run by little ones and zeroes, little bits of data. It's all just electrons. ...... There's a war out there, old friend. A world war. And it's not about who's got the most bullets. It's about who controls the information. What we see and hear, how we work, what we think... it's all about the information!

Well, you get the point. We all know that information is king and analyzing and visualizing that information is becoming more and more important. Presenting data in a way that is easy to digest is key to getting your audience to understanding that data. So let me make an introduction...

R is an open source environment for statistical computing. It can do some pretty neat breakdowns of your data and has a lot of built in functions for doing so. One of it's great strengths is generating production quality graphics and charts. This is what I needed it for and what I'll be explaining here in a moment. I learned R by watching a video introduction to R created by Decision Science News. There were 2 actually. But not very long and it got me to a base level. I then installed R on my Mac, it was cake. Go to the R site, download the DMG, run the R executable and you're ready to go. That got me up and running so I could start playing around on my own and using other examples from the web. YMMV on other platforms.

Now for the example. Let's set the stage. Say you have some data in a table, for example, a race my girlfriend competed in, the 2006 San Diego 10K race. I copied, pasted that data into a file, scrubbed it down, did some math with perl to get me the # of seconds, and ended up with a CSV file. Download the file, save it locally, read that file in with R:

race<-read.csv("race.csv")

You are reading that CSV file in as a table into a variable called 'race'. Because that CSV has a header as the first line, it automatically assigns variables based on those column names. To reference those columns, use 'race$CITY', to check out the 'CITY' column. So to check out what you've just done, type "race" on the console. Typing the variable name will spit it all back out. To see a breakdown of what that variable contains, type:

summary(race)

To see stats on the racers ages, type in:

 summary(race$AGE)

Which spits out:

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
10.00   28.00   35.00   37.06   44.00   81.00

Minimum age of a runner was 10, oldest was 81. Average age was 37.06 years old. Doing this for race$SEX shows us there were 411 women and 474 men. Neat! Now for the visuals:

Below is a script I used to generate the graph above. You can see how I am plotting the dots, and drawing both lines:

race<-read.csv("race.csv")

#Main Plot.
plot(race$SECONDS/60,race$AGE,
col="#5fae27",
main="",
xlab="Minutes",
ylab="Age",
cex=0.5,
type="p")
# Set the Title
title(main="Age vs Time")
# Draw the Red Line
lines(stats::lowess(race$SECONDS/60,race$AGE,f=0.1),
col="red",
lwd=2)
#Draw the Blue Line
lines(stats::lowess(race$SECONDS/60,race$AGE,f=0.3),
col="blue",
lwd=3)

Not too hard, not too much code...pretty easy in fact! One of the great things about R is the built in help. Any of those functions, just type:

?function

..and you'll have immediate help. I encourage you to do that for the example above, to better understand it. It will describe far better than I can how each one of those functions works.

Let's generate another one, a histogram. That's real easy, you can just type this:

hist(race$AGE,col="RED",xlab="Age",breaks=100,main="Histogram of Racers Age"

To generate this:

So, what did I learn from the creation of this plot? My initial suspicion was that younger people would do better in the race...the data shows that is's almost average across the board. The average age is in the late 30's, but the histogram shows the biggest group was mid-late 20's. Hardly anyone in their early 20's even entered the race...too busy drinking? Also, there is a neat little cluster at the bottom left of the plot that shows a group of young kids in their teens that did well.

I have been making more of these, mostly around sysadmin type stuff. I'll post those as I get more time.

I got Joe Jobbed!

noreply@blogger.com (Nathan) — Wed, 16 Apr 2008 02:37:00 +0000

In case you've never heard of this, getting 'Joe Jobbed' is when a spammer uses your e-mail address as the 'From:' in their spam and you get all the billion bounces. It rendered my Blackberry useless all afternoon, and I thought I'd finally be screwed on this one. Alas, SpamAssassin 3.2.0 and above (I'm running 3.2.4) comes with rules to block this. Unknown to me, most of the bounces were already going to my spam folder, but any that didn't have the spam body were getting to me. I followed the instructions on this page: http://wiki.apache.org/spamassassin/VBounceRuleset to make sure it was all set up and wouldn't classify bounces from my own server as bad. Then I set up the following procmail rule to put bounces in their own folder that I can review and run stats on:


:0:
* ^X-Spam-Status:.*ANY_BOUNCE_MESSAGE.*
bounces

Done!

More good reading on this: http://www.sitepoint.com/print/sabotage-coping-joe-job

UNIX paste, sed and nl commands

noreply@blogger.com (Nathan) — Wed, 27 Feb 2008 03:34:00 +0000

3 highly useful commands you will find as part of your standard UNIX toolbox. I'll give an example situation for relevance:

I quite often find myself needing to merge 2 files together for some reason or another. My latest awesomeness consists of ripping/encoding favorite seasons of DVD's I own so my MediaCenter can have an easily accessed library (that I can also stream to my iPod Touch). When the encoding is done, I get files based on the name of the DVD Media and the track number. Like this:

Blah Season 1 Disc 1-1.mp4
Blah Season 1 Disc 1-2.mp4
Blah Season 1 Disc 1-3.mp4
Blah Season 1 Disc 1-4.mp4
Blah Season 1 Disc 2-1.mp4
Blah Season 1 Disc 2-2.mp4
Blah Season 1 Disc 2-3.mp4
Blah Season 1 Disc 2-4.mp4

I can go to a place like Wikipedia or Amazon and find a list of the track names, that should look like this:

Pilot
The Fat Man
Little John
Howard
The Reconing
Half Way
Blah
Blah Pt 2

What I want to end up with is files named something like this:

Blah - S01E01 - Pilot.mp4
Blah - S01E02 - The Fat Man.mp4

It'd be so much easier to rename these on the command line if I could at least partly automate it. Re-typing is a PITA. So, here is how I do it to save a lot of time...

If I assume that they were ripped in order, I can get a listing of the order of the episode files based on the time stamps (ls) and add a " to the front and end of the line (sed commands):

# ls -tr *.mp4 | sed 's/^/"/g' | sed 's/$/"/g' > tracklist.txt

Next I go to Amazon, Wikipedia, whatever. Find a list of episodes that is represented with tables (which cut-n-pastes as individual lines with tabs as the delimiter) and paste it into a vi edit session:

# vi episodes.txt
In general, edit this file down to a point that the only thing that exists are the track names, one per line. Let's pretend the first column contained the name of the track, 2nd column the Writer, etc. We only care about the first column, so you can execute this command in vi:

:%s/\t.*//g

You can do other various cleanup like removing the " character:

:%s/"//g

Clean up other stuff like invalid shell characters, extra spaces, etc. This is the least automated part, but a hell of a lot faster/easier than re-typing. Especially if you are a vi whiz. If you use some other text editor, I'm sure this can be accomplished in a similar fashion.

Now we have 1 file that is the list of mp4 files, in order, and another that is the episode names, in order. What we now need is to make a file with what we actually want the files to be named. I accomplish this with the following command-line awesomeness:

nl -n rz -w 2 -s " - " episodes.txt \
| sed "s/^/\"Blah - S01E/g" \
| sed "s/$/.mp4\"/g" \
> newnames.txt

To break that down, this is what is happening:

The nl command adds numbered lines...the -n means "right justified, padded zeros" and the -w means "padded with 2 characters" and the -s means "separate the columns with what's in the quotes".
The 2 sed commands add a " to the beginning and end of the lines.
The > sends the output to a file.

Output looks like this:

"Blah - S01E01 - Pilot.mp4"
"Blah - S01E02 - The Fat Man.mp4"
"Blah - S01E03 - Little John.mp4"
"Blah - S01E04 - Howard.mp4"
"Blah - S01E05 - The Reconing.mp4"
"Blah - S01E06 - Half Way.mp4"
"Blah - S01E07 - Blah.mp4"
"Blah - S01E08 - Blah Pt 2.mp4"

Now we merge the 2 files and prepend the 'mv' command to get a script we can run:

paste tracklist.txt newname.txt | sed 's/^/mv /g' > script.sh

Which looks like this:

mv "Blah Season 1 Disc 1-1.mp4" "Blah - S01E01 - Pilot.mp4"
mv "Blah Season 1 Disc 1-2.mp4" "Blah - S01E02 - The Fat Man.mp4"
mv "Blah Season 1 Disc 1-3.mp4" "Blah - S01E03 - Little John.mp4"
mv "Blah Season 1 Disc 1-4.mp4" "Blah - S01E04 - Howard.mp4"
mv "Blah Season 1 Disc 2-1.mp4" "Blah - S01E05 - The Reconing.mp4"
mv "Blah Season 1 Disc 2-2.mp4" "Blah - S01E06 - Half Way.mp4"
mv "Blah Season 1 Disc 2-3.mp4" "Blah - S01E07 - Blah.mp4"
mv "Blah Season 1 Disc 2-4.mp4" "Blah - S01E08 - Blah Pt 2.mp4"

Check the script for sanity, then run it!

bash -x script.sh

w00t!

If you want the shell script I use to automate this somewhat:

#! /bin/sh

ls -tr *.mp4 | sed 's/^/"/g' | sed 's/$/"/g' > tracklist.txt
nl -n rz -w 2 -s " - " episodes.txt \
| sed "s/^/\"$1 - $2E/g" \
| sed "s/$/.mp4\"/g" \
> newnames.txt
paste tracklist.txt newnames.txt | sed 's/^/mv /g' > script.sh

The arguments are the name of the series and the season, like this:

bash ./rename.sh Blah S01

P.S. If you're trying to guess the show by the track names, I made them up. :)

My Synergy Setup

noreply@blogger.com (Nathan) — Tue, 26 Feb 2008 03:29:00 +0000

Synergy rules. I've been talking about it for a few weeks now, expounding on it's virtues over other options like x2vnc. A friend asked that I document my setup...here ya go Eddy: :)

My current install of synergy runs on on my Mac Pro, my linux box (centos5) and my MacBook Pro laptop. The Mac Pro is the controlling keyboard and reaches across both of the other computers. The monitor layout is the Mac Pro in front (2 monitors), the Linux box directly to the left and the laptop below the linux box.

After I downloaded synergy, I placed the 2 binaries (synergys,synergyc) in /usr/local/bin on my Mac. If the dir doesen't exist, create it. I also created /usr/local/etc for the config files.

Config on the Mac Pro:


n8pro:bin nathan$ cd /usr/local/bin
n8pro:bin nathan$ ls
jhead  jpegtran synergyc synergys xv

The config file on the Mac Pro:


n8pro:~ nathan$ cd /usr/local/etc/
n8pro:etc nathan$ cat synergy.conf
  section: screens
     n8pro.local:
     n8bookpro.local:
     homelinux:
  end
  section: links
     n8pro.local:
         left = homelinux
     homelinux:
         right = n8pro.local
         down = n8bookpro.local
n8bookpro.local:
   up = homelinux
  end

Commands I run on the Mac Pro (server):

# synergys -f -c /usr/local/etc/synergy.conf

Commands I run on the Linux Box (client):

# synergyc -f n8pro

Commands I run on the MacBook Pro (client):

# synergyc -f n8pro

Extra credit:

I also use this at work. A Linux box is the control. Synergyc will re-try connections every 30 sec or so, which means I can launch synergyc clients and let them fail when I'm not on the network. This means I can run it once and connect/disconnect from work/home networks and have seamless mouse sharing with very little work required. To set this up on the Linux box, i had a config similar to the one above for the Mac Pro. On the laptop I have one script that I run that fires off both synergyc clients at the same time. They stay running until a reboot. After reboot, I run it again and I'm set. This is the script:

cat /usr/local/bin/setup_synergy.sh
#! /bin/bash
killall synergyc
sleep 2
synergyc -f n8pro &
synergyc -f n8linuxbox &

Restoring lost screen sockets

noreply@blogger.com (Nathan) — Thu, 20 Dec 2007 11:16:00 +0000

Do your screen sockets get nuked and you can't connect to your screen sessions? If something auto-cleans the /tmp dir, you may have seen this before. If you're like me, you end up with 10 or so shells with various programs running that you then have to follow the process tree to kill off. This is my only real (huge) frustration with screen. Well, here's ya go:

ps -x | grep SCREEN | awk '{print $1}' | xargs kill -CHLD

Enjoy!

Christmas tree decorating at high speed

noreply@blogger.com (Nathan) — Wed, 05 Dec 2007 22:37:00 +0000

Christmas Tree 2007 from nathan hubbard on Vimeo.

If you watch closely, you can see some sillyness at the beginning. Hehe. Good stuff!

Peyton Lost, boooooo hoooooo

noreply@blogger.com (Nathan) — Mon, 12 Nov 2007 07:15:00 +0000

I don't even like football, but I did tonight. >:-)

Go Chargers!!

Spiders on Drugs

noreply@blogger.com (Nathan) — Thu, 08 Nov 2007 19:48:00 +0000

I guess everyone has seen this but me:

http://youtube.com/watch?v=sHzdsFiBbFc

In the 1960s, Dr. Peter Witt gave drugs to spiders and observed their effects on web building. This short film about the results of the experiment was created by First Church Of Christ, Filmmaker.

looking at your boot sector

noreply@blogger.com (Nathan) — Tue, 24 Jul 2007 09:52:00 +0000

Yes yes, I know, sometimes you just get curious about your boot sector. Your Sysadmin says you'll go to hell if you play with it, and god knows what would happen if you tried to modify it. But sometimes, when you've been screwing with Lilo and Grub and you can't really remember which one is installed, it's nice to have a peek:

dd if=/dev/hda of=mbr.img bs=512 count=1 xxd -g 4 mbr.img

Where /dev/hda is the drive you want to look at.

Output should look something like this:
-snip- 0000150: 31fffcf3 a51f61ff 26427cbe 7f7de840 1.....a.&B|..}.@ 0000160: 00eb0ebe 847de838 00eb06be 8e7de830 .....}.8.....}.0 0000170: 00be937d e82a00eb fe475255 42200047 ...}.*...GRUB .G 0000180: 656f6d00 48617264 20446973 6b005265 eom.Hard Disk.Re 0000190: 61640020 4572726f 7200bb01 00b40ecd ad. Error....... 00001a0: 10ac3c00 75f4c300 00000000 00000000 ..<.u........... -snip-
Oh no, I've displayed all my most intimate parts for everybody to see. Oh well...hey, there's Grub. Cool. :)

1:1 NAT on your Linux/Netfilter Firewall

noreply@blogger.com (Nathan) — Wed, 11 Jul 2007 05:55:00 +0000

So you want to map a public IP to a private IP behind your Linux (netfilter) based firewall. Here is the syntax:


## Standard Stuff ##
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables --table nat -A POSTROUTING -o eth0 -j SNAT --to $NAT_IP

## The 1:1 NAT stuff ##
iptables -t nat -A PREROUTING -d $EXTERNAL_IP  \
        -j DNAT --to-destination $INTERNAL_IP
iptables -t nat -A POSTROUTING -s $INTERNAL_IP \
        -j SNAT --to-source $EXTERNAL_IP
iptables -t nat -A POSTROUTING -s $INTERNAL_NET -d $INTERNAL_IP \
        -j SNAT --to-source $NAT_IP

Where $NAT_IP is the external IP of your firewall. The last rule is required if you want hosts on your internal net to be able to talk to that external IP as well.

MachDB

noreply@blogger.com (Nathan) — Thu, 14 Jun 2007 10:00:00 +0000

After a successful night of coding, I decided to start blogging my development of the MachDB.

Yay! First post! I'll describe it more later.

(also checking out the cool labels that blogger has now)

And going to bed.

Great song: Two Sisters by Fiction Plane

noreply@blogger.com (Nathan) — Sun, 27 May 2007 08:08:00 +0000

Watching Jay Leno this week, I heard a band I wasn't familiar with, Fiction Plane. 3 talented guys sang a great song called "Two Sisters". One of the best rock/pop songs I've heard in a long time. And going back to look up the name, I discovered that the NBC Jay Leno web site has a free 160kbps MP3 for download....right above a link to buy it. :) Get it right here, before somebody realizes the (what has to be) mistake and takes it down.

I'm always impressed with bands that can create great music with only a few bodies. Trivia note: If you look them up, you may think the lead singer, Joe Sumner, looks familiar. Sting is his dad.

1080p Sky Timelapse

noreply@blogger.com (Nathan) — Sat, 26 May 2007 09:21:00 +0000

I got a new toy recently. A Canon TC-80N3...a programmable remote timer for my DSLR. I've always been a huge fan of Time Lapse video, especially the clouds moving and flowers blooming. I made one of my own, recorded at full image size and scaled *down* to 1080p for your viewing pleasure. File is 28MB: Mira Mesa Night Sky Timelapse 2007-05-09. Right-click & save, then play. Flash preview below:

Other than seeing the stars move at 450 times normal speed, the neatest part is the planes flying around at the bottom. They almost look like UFO's. You can see the fog roll in later in the shots as the camera gets a layer of dew on the lens. The exposure was 10sec each, taken every 30 seconds and playback is at 15fps.

I'll be making more of these and posting them to my blog as I have time. Tell me what you think.

Other timelapse posts:

Blogerize My Time Lapse
6 hours of Friday
More Timelapses
Pumpkins
Flight of the Servers/VMIX Down Page