Archive for September, 2007

Removing Duplicate Files from iTunes

Saturday, September 29th, 2007

While iTunes has its issues, for the most part it makes for a fairly passable media organizer. As long as I let it do its job, I’m pretty happy with how it handles the files on the backend. When I moved my iTunes library to a new external drive, however, I came across a pretty egregious “bug” that left me with thousands of duplicate files and no good way to remove them.

By default, iTunes stores files in “/Users//Music/iTunes/iTunes Music”. When you drag and drop a file from your desktop, or wherever, it copies the file into this directory, tossing it into a subdirectory based on Artist and Album name and renaming it according to the track info you specify in iTunes. When you change the Album name or Track name, the file is is updated and moved automatically, so the location and name of the file always reflect what you see in iTunes. Personally, I like this behavior.

I didn’t run into problems until I attempted to move my files from the default directory to a directory on my new drive, a 500G firewire affair. At first, everything worked properly when I changed the settings; it took while, but it slowly copied all the the files to the new location, “/Data/iTunes”, and once it was done, everything was great. And everything remained great until I made the mistake of opening iTunes while my new drive was unmounted.

Little did I know that when iTunes couldn’t find “/Data/iTunes”, the new directory, it changed itself back to its default: “/Users//Music/iTunes/iTunes Music”. Upon finding all the files still there, it quickly settled in and changed all the entries in the database to point to these files. When I finally realized what had happened, I remounted the new drive and changed the iTunes directory again- and that’s when all my troubles began.

iTunes once again began to copy my files to new the drive- only this time, the files already existed. Rather than simply use the existing files, it created new copies, appended with ” 1″, and I was left with several thousand duplicate files and no easy way to remove them.

To deal with this issue, I wrote a simple PHP script to crawl through my iTunes directory and deal with files that end in ” 1″. In the case where both files exist (with and without ” 1″), the script compares the two files. If they are identical, it tries to rectify the situation by removing the old file and re-adding the remaining file to iTunes using osascript, OS X’s command line Applescript tool. Adding the file triggers iTunes to automatically rename the file properly, thus removing the ” 1″. By default, you’re prompted before any file is deleted. You can disable this by hitting ‘a’ at the prompt.

In the case where the two files are not equal, they are skipped. You’re going to have to deal with those manually. Sorry. It also has problems with files that contain special characters. You’ll have to deal with those as well… but come on- how many Blue Öyster Cult, Mötley Crüe or Björk songs do you have, anyway?

In the case where only one file exists, it will simply re-add the file to iTunes, on the theory that iTunes will fix it if the name is wrong, and ignore it otherwise.



* Name: fix_itunes_dupes.php
* Author: patrick
* Usage: fix_itunes_dupes.php
* This script is provided AS-IS.  No warranty is either expressed or implied.
* Use at your own risk.
* Feel free to use, modify, duplicate, or take credit.
// Set this variable to your iTunes directory.
$itunes_dir = "/Users/{$_ENV['USER']}/Music/iTunes/iTunes Music";

if (!file_exists($itunes_dir)) {
    exit("{$itunes_dir} does not exist.");

// Collect itunes files that end in " 1".
$find = "find \\"{$itunes_dir}\\" -type f -name \\"* 1.*\\"";
$files = explode("\n", `$find`);
$file_count = count($files) - 1;
// 'find' comes with a bonus carriage return.  Get rid of it or try to remove
// '.' (that's bad).
// Set some defaults.
$all = false;
$count = 0;

// Loop through the files.
foreach ($files as $file1) {
    print "Processing {$count}/{$file_count}:\n";
    // Skip Movies and TV Shows.
    if (preg_match("/\/Movies\//", $file1) || preg_match("/\/TV Shows\//", $file1)) {
        print "Skipping TV Show or Movie:\n";
        print "{$file1}\n";
    } else {
        preg_match("/^(.+) [0-9].([^.]+)$/", $file1, $matches);
        $file = "{$matches[1]}.{$matches[2]}";
        if (file_exists($file)) {
            $md5 = md5(file_get_contents($file));
            $md51 = md5(file_get_contents($file1));
            print "{$file} ({$md5})\n";
            print "{$file1} ({$md51})\n";
            if ($md5 == $md51) {
                // If the two files are identical, we can space one of them.
                print "{$file} = {$file1}\n";
                if (!$all) {
                    // Options are:
                    //  y - delete the file. default.
                    //  n - skip this file.
                    //  a - stop asking, just do them all.
                    //  q - quit.
                    print "Delete? [Y/n/a/q]: ";
                    $char = substr(fgets(STDIN), 0, 1);
                if ($char == "q") {
                } elseif ($char != "n") {
                    if ($char == "a") {
                        $all     = true;
                    print "Deleting {$file}\n";
            } else {
                print "Files do not match.  Skipping.\n";
        } else {
            // There is only one file; go ahead and add it to iTunes-
            // shouldn't hurt.
            print "{$file1} exists on its own.\n";
    print "\n";

function add2itunes($file) {
    $file = preg_replace("/\//", ":", $file);
    // Please forgive the toothpicks- numerous files have 's.
    $command = "osascript -e \\"tell application \\\\"iTunes\\\\" to add file \\\\"{$file}\\\\" to playlist \\\\"Library\\\\" of source \\\\"Library\\\\"\\"\n";
    print $command;


Sample Output

Processing 31/39:
/Data/iTunes/The Beastie Boys/Licensed To Ill/13 Time to Get Ill.m4a (42cbc19b9d2feb83899201793ee1e00e)
/Data/iTunes/The Beastie Boys/Licensed To Ill/13 Time to Get Ill 1.m4a (e2f49dfc12db5fb8f8d3954b9671870c)
Files do not match.  Skipping.
Processing 32/39:
/Data/iTunes/The Bloodhound Gang/One Fierce Beer Coaster/07 Asleep at the Wheel.mp3 (6d12ce65c4cba6748224e2262feba67a)
/Data/iTunes/The Bloodhound Gang/One Fierce Beer Coaster/07 Asleep at the Wheel 1.mp3 (6d12ce65c4cba6748224e2262feba67a)
/Data/iTunes/The Bloodhound Gang/One Fierce Beer Coaster/07 Asleep at the Wheel.mp3 = /Data/iTunes/The Bloodhound Gang/One Fierce Beer Coaster/07 Asleep at the Wheel 1.mp3
Delete? [Y/n/a/q]: y
Deleting /Data/iTunes/The Bloodhound Gang/One Fierce Beer Coaster/07 Asleep at the Wheel.mp3
osascript -e "tell application \"iTunes\" to add file \":Data:iTunes:The Bloodhound Gang:One Fierce Beer Coaster:07 Asleep at the Wheel 1.mp3\" to playlist \"Library\" of source \"Library\""
file track id 53184
Processing 33/39:
Skipping TV Show or Movie:
/Data/iTunes/TV Shows/Battlestar Galactica/3-03 Exodus_ Part 1.mp4

Video Server, Part 4: Converting

Monday, September 24th, 2007

This is Part Four of my series on building a video server using Apple’s iTunes/AppleTV combination. This article details the process of converting existing video files to a compatible format.

When I initially started this whole video thing, my goal was to get video on my iPod, so naturally one of the applications I came across when I was searching for tools to convert my collection of .avi files was the Videora iPod Converter by Red Kawa (v2.25). Though billed as an iPod converter, this front-end for ffmpeg allows you to set the output values to almost anything, making it ideal for converting an existing .avi file to any flavor of .mp4 you’d like.

At first I was put off by the ad-thick interface and the hideous design, but once I got past that I found a fairly full featured, functional application. Day to day usage is trivial, as the program is built around a “One-Click” queue that functions pretty much as intended, but getting it configured took a bit of work.

Step 1: Set the Video Output Folders.
After you start the program, choose “Settings” from the menu at the top of the screen. Chose “Devices” from the tabs menu that appears, then “iPod” from Device Settings. You have two fields for Video Output Folders: MPEG-4 Videos and H.264 Videos. I don’t know which is which, so I set both of them.

Fig 1: Set the Video Output Folders.

Step 2: Set the Encoding Profile and One-Click settings.
Change the tab from “Devices” to “Converter”. In the “Encoding Profiles” box, set Device to iPod, and Profile to “H.264 VGA 1024 kbps Stereo/160kbps”. In the “One-Click” box, set “Device” and “Profile”. I also set “Process Priority” to “High”, but this is the only thing I use this computer for.

Fig 2: Set the Encoding Profile and One-Click settings.

Step 3: Edit Profile, set General Properties.
Click on “Edit Profile”. Leave Name set to “H.264 VGA 1024 kbps Stereo/160kbps”, and change Encoder to “AppleTV 1-Pass – FFmpeg MINB”, if it’s not already set.

Fig 3: Edit Profile, set General Properties.

Step 4: Edit Profile, set Video Properties.
Click the “Video” option in the menu, and set the options as they appear in Fig 4. For the most part, I prefer to leave the resolution and aspect ratio alone, and make sure Auto Resize is not enabled. However, do make sure to set the Framerate value to “29.97”. A number of applications and hardware decoders can be touchy about this value.

Fig 4: Edit Profile, set Video Properties.

Step 5: Edit profile, set Audio Properties.
Click the “Audio” icon in the top menu, and compare the settings to Fig 5. To be perfectly honest, I don’t remember whether or not I changed any of the defaults, or if they matter.

Fig 5: Edit Profile, set Audio Properties

Step 6: Convert.
Once the profile has been configured, converting is a breeze. In the top menu click “Convert” and choose the “Current Conversion” tab. Click the “One-Click Convert” button to choose a list of .avi files. They will automatically be added to the queue and conversion will begin immediately. Completed files will be dropped in the Video Output Folders defined in Step 1, and should be fully iTunes compatible.

Fig 6: Convert.

Your mileage will vary, of course, but my conversions are usually a tad faster than realtime. A 42 minute file, for example, will usually take 30 to 40 minutes to complete, and the quality, while not stellar, is adequate. There are likely a number of places where this could be improved.

In the next article in this series, I will present the scripts and techniques I use to add meta-data to the video files, including plot descriptions, artwork and ratings.

Fixing a PHP CLI segmentation fault

Monday, September 17th, 2007

Another Short Answer to a Specific Question:

Sometimes, after an upgrade, any PHP script run from the command line causes a segmentation fault, even though the script seems to run fine. For example:

[03:13:01 argon:~]$ php -v
PHP 5.2.4 with Suhosin-Patch (cli) (built: Sep 17 2007 02:28:25)
Copyright (c) 1997-2007 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2007 Zend Technologies
Segmentation fault: 11 (core dumped)
[03:13:02 argon:~]$

Whenever this happens, I can usually resolve the problem by changing the order of the modules listed in extensions.ini. Three times out of four, moving “” to the top of the file fixes the problem.

Run ‘screen’ and reconnect to the same session without arguments

Sunday, September 16th, 2007

This is just a little nugget I’ve had around for a while. I’m dusting it off and posting here more to test the WP-Syntax WordPress Plugin than any other reason.

I use screen, but I don’t leave myself logged in when I’m not at my computer. I can usually type ‘screen -R’ to reconnect to the same session the next day, but if I have more than one screen session running at once, I need to ‘screen -ls’, then figure out which one to reattach to. At least I think I need to do that… in addition to being to lazy to do all that, I’m too lazy to scour the man pages looking for the way to make it create or reattach to the same session every time.

If you call ‘screen’ with parameters, they will be used instead… This should allow you to use screen completely normally when necessary.

# Rename your screen binary to something else to get it out of
# the way, and save this script as 'screen' into the same
# directory. Make a note of what you called the original here:

# Make $DEFAULT_NAME whatever you want; you'll never actually see it:

if [ "$#" -eq 0 ]; then
    if [ "$SCREENLS" = "" ]; then
        $SCREEN -r "$DEFAULT_NAME"
    $SCREEN $@

Video Server, Part 3: Encoding

Friday, September 14th, 2007

This is the third in a series of articles that will detail the tools and methods I’m using for my personal media server at home. Part 3: Encoding gets to the meat of the matter, and finally addresses how to turn the DVD into a video file. Future articles will detail converting, integration and disaster management.

Encoding is where all magic happens. The data on the DVD is finally converted to a standalone file to be used in other applications. This is also where most of the confusion lay, where incompatibilities creep in, and where most of the time is spent waiting for the computer to finish something. This time spent waiting is also what makes trial and error such a bitch; depending on your setup, you may have to wait as long as four hours to find out if the file you generated is any good, or if it will work in with your chosen application.

Right up front, let me say that I’m no expert on this subject. I had one goal: decent quality movies at a reasonable size that worked in iTunes. There are a ton of videophiles out there who argue about this codec or that codec, and Dolby digital sound vs. whatever regular old sound is called, but when it comes down to it, I’m interested in convenience and watchability above all else. That being said, though, this method produces a file that’s indistinguishable from DVD, at least to my eyes.

One more caveat, before I delve into the minutiae. I’m going to go through all of the options I set for encoding, but there are a number of places where I didn’t make a conscious decision to do it one way or the other, and I don’t know if changing it will make any difference. For example, in one case I turn auto-cropping on, because I suspected that leaving it off would give me black bars along the top and bottom of my image. I never tried changing it, though, because the settings I’m using give me the results I want.

I played around with a number of different packages on both Mac and PC before I finally ended up using Handbrake v0.9.0 on the PC, a free, open-source, cross-platform package that used to be exclusively for the Mac, but eventually went both ways. It’s not super-fast, but it produces an excellent quality video and is fully operational from the command line; in fact, from what I can tell the GUI is just tacked on to make it easier to build the command line parameter list. In the interest of gratuitous multi-media, however, I’m going to go through configuration using the GUI before recommending you abandon it altogether.

Step 1: Choose your source directory.

The first step, once you start the program, is to choose the directory where you stored your DVD rip. You can find more information on ripping in Part 2 of this series. This example is based on where the files would be after following that tutorial, or F:\FullDisc\JURASSICDTS\VIDEO_TS\.

Fig 1: Choose your source files.

Step 2: Choose AppleTV from the presets:

From the menu bar, choose AppleTV from the preset menu. This will set a number of options.

Fig 2: Choose AppleTV from the presets.

Step 3: Choose the destination and cropping options.
When you specify the destination file through the Browse menu, Handbrake will automatically change the file extension to .m4v, the default file extension for Apple’s movie files. Since a number of my post-encoding scripts were written to use .mp4 files, and I have hundreds of other files with that extension, I change it back to .mp4. This seems to work fine. Under the “Picture Settings” tab, set Cropping to “Auto Cropping”.

Fig 3: Set the destination file and the cropping.

Step 4: Change the bitrate.
Change to the “Video Settings” tab and change the bitrate to 1024. I’ve found this bitrate produces excellent results and a file size of around a Gig for most movies. As far as I know, the value 1024 has no special meaning in this context- it just seemed computer-y.

Fig 4: Change the bitrate.

Step 5: Ditch the GUI.
The last tab, “Query Editor”, has button labeled “Generate Query Now”, which will give you the command line options based on your settings. I recommend you copy this text and run Handbrake from the command line:

C:\Program Files\Handbrake\hbcli.exe -i "F:\FullDisc\JURASSICDTS\VIDEO_TS" -o "E:\Upload\Jurassic Park.mp4" -e x264 -E faac -p -m -b 1024 -x bframes=3:ref=1:subme=5:me=umh:no-fast-pskip=1:trellis=2 -B 160 -R 48

… or, if you’re running Cygwin, which I also recommend:

/cygdrive/c/Program\ Files/Handbrake/hbcli.exe -i "F:\FullDisc\JURASSICDTS\VIDEO_TS" -o "E:\Upload\Jurassic Park.mp4" -e x264 -E faac -p -m -b 1024 -x bframes=3:ref=1:subme=5:me=umh:no-fast-pskip=1:trellis=2 -B 160 -R 48

Fig 5: Generate the query if you want to run it from the command line instead

At this point you’ve got a nice file that you can drag into iTunes and watch on your AppleTV. A word of caution, though: on my machine, this process will occasionally fail with a segmentation fault after two to four DVDs. Rebooting and restarting the process works for me.

In the next entry in this series, I will go deal with converting existing files into MP4 files, mostly for dealing with downloaded TV Shows, and after that I will address adding meta-data, such as artwork and plot descriptions, to make your videos pretty. Stay tuned for that- there will be scripts.

No Thumbnail for an Uploaded Image in WordPress

Thursday, September 13th, 2007

A Short Answer to a Specific Question:

If you upload an image in WordPress, and you don’t get the option to “Show Thumbnail” in the “Send to Editor”, it’s probably because the image was too big.

Also: there is no good way to resize an image in iPhoto.

A free tool to resize images on the Mac is ImageWell, from XtraLean Software.

Short Answers to Specific Questions

Thursday, September 13th, 2007

I’ll be perfectly frank; I don’t like blogs. For the most part, I think they’re pretentious, self centered, and everytime someone references “my audience” when they’re not talking about a show they were headlining at Shoreline, I have the mild urge to cram a squirrel in ’em.

That being said, though, I’ve often thought that if I were ever to start a blog, in addition to whatever hare-brained reason I had for doing it in the first place, I would also try to add a little bit to the sum total of human knowledge by occasionally posting the one thing I’m always grateful for when I’m lucky enough to find it: the short answer to the specific question.

From now on, when I’m searching for one specific piece of information, or the answer to a very direct technical question, I’ll try to post it here, so on the off chance that someone else is looking for it, they might stumble on it. You never know.