Tuesday, February 26, 2008
UNIX paste, sed and nl commands
3 highly useful commands you will find as part of your standard UNIX toolbox. I'll give an example situation for relevance:
I quite often find myself needing to merge 2 files together for some reason or another. My latest awesomeness consists of ripping/encoding favorite seasons of DVD's I own so my MediaCenter can have an easily accessed library (that I can also stream to my iPod Touch). When the encoding is done, I get files based on the name of the DVD Media and the track number. Like this:
I can go to a place like Wikipedia or Amazon and find a list of the track names, that should look like this:
What I want to end up with is files named something like this:
It'd be so much easier to rename these on the command line if I could at least partly automate it. Re-typing is a PITA. So, here is how I do it to save a lot of time...
If I assume that they were ripped in order, I can get a listing of the order of the episode files based on the time stamps (ls) and add a " to the front and end of the line (sed commands):
Next I go to Amazon, Wikipedia, whatever. Find a list of episodes that is represented with tables (which cut-n-pastes as individual lines with tabs as the delimiter) and paste it into a vi edit session:
# vi episodes.txt
In general, edit this file down to a point that the only thing that exists are the track names, one per line. Let's pretend the first column contained the name of the track, 2nd column the Writer, etc. We only care about the first column, so you can execute this command in vi:
You can do other various cleanup like removing the " character:
Clean up other stuff like invalid shell characters, extra spaces, etc. This is the least automated part, but a hell of a lot faster/easier than re-typing. Especially if you are a vi whiz. If you use some other text editor, I'm sure this can be accomplished in a similar fashion.
Now we have 1 file that is the list of mp4 files, in order, and another that is the episode names, in order. What we now need is to make a file with what we actually want the files to be named. I accomplish this with the following command-line awesomeness:
To break that down, this is what is happening:
The nl command adds numbered lines...the -n means "right justified, padded zeros" and the -w means "padded with 2 characters" and the -s means "separate the columns with what's in the quotes".
The 2 sed commands add a " to the beginning and end of the lines.
The > sends the output to a file.
Output looks like this:
Now we merge the 2 files and prepend the 'mv' command to get a script we can run:
Which looks like this:
Check the script for sanity, then run it!
w00t!
If you want the shell script I use to automate this somewhat:
The arguments are the name of the series and the season, like this:
P.S. If you're trying to guess the show by the track names, I made them up. :)
I quite often find myself needing to merge 2 files together for some reason or another. My latest awesomeness consists of ripping/encoding favorite seasons of DVD's I own so my MediaCenter can have an easily accessed library (that I can also stream to my iPod Touch). When the encoding is done, I get files based on the name of the DVD Media and the track number. Like this:
Blah Season 1 Disc 1-1.mp4
Blah Season 1 Disc 1-2.mp4
Blah Season 1 Disc 1-3.mp4
Blah Season 1 Disc 1-4.mp4
Blah Season 1 Disc 2-1.mp4
Blah Season 1 Disc 2-2.mp4
Blah Season 1 Disc 2-3.mp4
Blah Season 1 Disc 2-4.mp4
I can go to a place like Wikipedia or Amazon and find a list of the track names, that should look like this:
Pilot
The Fat Man
Little John
Howard
The Reconing
Half Way
Blah
Blah Pt 2
What I want to end up with is files named something like this:
Blah - S01E01 - Pilot.mp4
Blah - S01E02 - The Fat Man.mp4
It'd be so much easier to rename these on the command line if I could at least partly automate it. Re-typing is a PITA. So, here is how I do it to save a lot of time...
If I assume that they were ripped in order, I can get a listing of the order of the episode files based on the time stamps (ls) and add a " to the front and end of the line (sed commands):
# ls -tr *.mp4 | sed 's/^/"/g' | sed 's/$/"/g' > tracklist.txt
Next I go to Amazon, Wikipedia, whatever. Find a list of episodes that is represented with tables (which cut-n-pastes as individual lines with tabs as the delimiter) and paste it into a vi edit session:
# vi episodes.txt
In general, edit this file down to a point that the only thing that exists are the track names, one per line. Let's pretend the first column contained the name of the track, 2nd column the Writer, etc. We only care about the first column, so you can execute this command in vi:
:%s/\t.*//g
You can do other various cleanup like removing the " character:
:%s/"//g
Clean up other stuff like invalid shell characters, extra spaces, etc. This is the least automated part, but a hell of a lot faster/easier than re-typing. Especially if you are a vi whiz. If you use some other text editor, I'm sure this can be accomplished in a similar fashion.
Now we have 1 file that is the list of mp4 files, in order, and another that is the episode names, in order. What we now need is to make a file with what we actually want the files to be named. I accomplish this with the following command-line awesomeness:
nl -n rz -w 2 -s " - " episodes.txt \
| sed "s/^/\"Blah - S01E/g" \
| sed "s/$/.mp4\"/g" \
> newnames.txt
To break that down, this is what is happening:
The nl command adds numbered lines...the -n means "right justified, padded zeros" and the -w means "padded with 2 characters" and the -s means "separate the columns with what's in the quotes".
The 2 sed commands add a " to the beginning and end of the lines.
The > sends the output to a file.
Output looks like this:
"Blah - S01E01 - Pilot.mp4"
"Blah - S01E02 - The Fat Man.mp4"
"Blah - S01E03 - Little John.mp4"
"Blah - S01E04 - Howard.mp4"
"Blah - S01E05 - The Reconing.mp4"
"Blah - S01E06 - Half Way.mp4"
"Blah - S01E07 - Blah.mp4"
"Blah - S01E08 - Blah Pt 2.mp4"
Now we merge the 2 files and prepend the 'mv' command to get a script we can run:
paste tracklist.txt newname.txt | sed 's/^/mv /g' > script.sh
Which looks like this:
mv "Blah Season 1 Disc 1-1.mp4" "Blah - S01E01 - Pilot.mp4"
mv "Blah Season 1 Disc 1-2.mp4" "Blah - S01E02 - The Fat Man.mp4"
mv "Blah Season 1 Disc 1-3.mp4" "Blah - S01E03 - Little John.mp4"
mv "Blah Season 1 Disc 1-4.mp4" "Blah - S01E04 - Howard.mp4"
mv "Blah Season 1 Disc 2-1.mp4" "Blah - S01E05 - The Reconing.mp4"
mv "Blah Season 1 Disc 2-2.mp4" "Blah - S01E06 - Half Way.mp4"
mv "Blah Season 1 Disc 2-3.mp4" "Blah - S01E07 - Blah.mp4"
mv "Blah Season 1 Disc 2-4.mp4" "Blah - S01E08 - Blah Pt 2.mp4"
Check the script for sanity, then run it!
bash -x script.sh
w00t!
If you want the shell script I use to automate this somewhat:
#! /bin/sh
ls -tr *.mp4 | sed 's/^/"/g' | sed 's/$/"/g' > tracklist.txt
nl -n rz -w 2 -s " - " episodes.txt \
| sed "s/^/\"$1 - $2E/g" \
| sed "s/$/.mp4\"/g" \
> newnames.txt
paste tracklist.txt newnames.txt | sed 's/^/mv /g' > script.sh
The arguments are the name of the series and the season, like this:
bash ./rename.sh Blah S01
P.S. If you're trying to guess the show by the track names, I made them up. :)
Monday, February 25, 2008
My Synergy Setup
Synergy rules. I've been talking about it for a few weeks now, expounding on it's virtues over other options like x2vnc. A friend asked that I document my setup...here ya go Eddy: :)
My current install of synergy runs on on my Mac Pro, my linux box (centos5) and my MacBook Pro laptop. The Mac Pro is the controlling keyboard and reaches across both of the other computers. The monitor layout is the Mac Pro in front (2 monitors), the Linux box directly to the left and the laptop below the linux box.

After I downloaded synergy, I placed the 2 binaries (synergys,synergyc) in /usr/local/bin on my Mac. If the dir doesen't exist, create it. I also created /usr/local/etc for the config files.
Config on the Mac Pro:
The config file on the Mac Pro:
Commands I run on the Mac Pro (server):
Commands I run on the Linux Box (client):
Commands I run on the MacBook Pro (client):
Extra credit:
I also use this at work. A Linux box is the control. Synergyc will re-try connections every 30 sec or so, which means I can launch synergyc clients and let them fail when I'm not on the network. This means I can run it once and connect/disconnect from work/home networks and have seamless mouse sharing with very little work required. To set this up on the Linux box, i had a config similar to the one above for the Mac Pro. On the laptop I have one script that I run that fires off both synergyc clients at the same time. They stay running until a reboot. After reboot, I run it again and I'm set. This is the script:
My current install of synergy runs on on my Mac Pro, my linux box (centos5) and my MacBook Pro laptop. The Mac Pro is the controlling keyboard and reaches across both of the other computers. The monitor layout is the Mac Pro in front (2 monitors), the Linux box directly to the left and the laptop below the linux box.

After I downloaded synergy, I placed the 2 binaries (synergys,synergyc) in /usr/local/bin on my Mac. If the dir doesen't exist, create it. I also created /usr/local/etc for the config files.
Config on the Mac Pro:
n8pro:bin nathan$ cd /usr/local/bin
n8pro:bin nathan$ ls
jhead jpegtran synergyc synergys xv
The config file on the Mac Pro:
n8pro:~ nathan$ cd /usr/local/etc/
n8pro:etc nathan$ cat synergy.conf
section: screens
n8pro.local:
n8bookpro.local:
homelinux:
end
section: links
n8pro.local:
left = homelinux
homelinux:
right = n8pro.local
down = n8bookpro.local
n8bookpro.local:
up = homelinux
end
Commands I run on the Mac Pro (server):
# synergys -f -c /usr/local/etc/synergy.conf
Commands I run on the Linux Box (client):
# synergyc -f n8pro
Commands I run on the MacBook Pro (client):
# synergyc -f n8pro
Extra credit:
I also use this at work. A Linux box is the control. Synergyc will re-try connections every 30 sec or so, which means I can launch synergyc clients and let them fail when I'm not on the network. This means I can run it once and connect/disconnect from work/home networks and have seamless mouse sharing with very little work required. To set this up on the Linux box, i had a config similar to the one above for the Mac Pro. On the laptop I have one script that I run that fires off both synergyc clients at the same time. They stay running until a reboot. After reboot, I run it again and I'm set. This is the script:
cat /usr/local/bin/setup_synergy.sh
#! /bin/bash
killall synergyc
sleep 2
synergyc -f n8pro &
synergyc -f n8linuxbox &
Subscribe to Posts [Atom]