Download

Pivot tables (also called contingency tables and cross tabulation tables) are a powerful means of data visualization and data summarization. When dealing with large data sets with multiple variables, or multiple datasets manually manipulating the pivot tables in WYSIWYG (what you see is what you get) spreadsheets can quickly become troublesome and error prone. In these instances it becomes preferred or even necessary to use a YAFIYGI (you ask for it you got it) model to automate all or part of the data summarization process.
There are already existing Python pivot table modules available. The ones I have found don't support multidimensional data, require Windows and Excel, or are incomplete and abandoned. They also are usually tailored towards an information technology audience as opposed to a scientific/research audience. On the other extreme are projects like PyTables. PyTables is an impressive undertaking but many datasets just aren't complex enough to justify the effort required to get data into PyTables. The pyvttbl module presented here offers a solution for datasets of "Goldilocks" complexity; too much for spreadsheets, but too little for coding custom solutions or configuring PyTables.

Platypus is a developer tool for the Mac OS X operating system. It can be used to create native, flawlessly integrated Mac OS X applications from interpreted scripts such as shell scripts or Perl, Ruby and Python programs. This is done by wrapping the script in an application bundle directory structure along with an executable binary that runs the script.

Platypus makes it easy for you to share your scripts and programs with those unfamiliar with the command line interface, without any knowledge of the Mac OS X APIs -- a few clicks and you will have your own Mac OS X graphical program. Creating installers, maintenance applications, login items, launchers, automations and droplets is very easy using Platypus.

Supports shell scripts, Perl, Python, PHP, Ruby, Expect, Tcl, AppleScript
Supports arbitrary interpreter specification
Executing scripts with root privileges via Apple's Security Framework
Drag and drop files or text snippets, which are then passed to the script as arguments
Can display graphical feedback of script execution as progress bar, text window with script output, WebKit HTML rendering or status item menus
Can create applications which run in the background (LSUIElement)
Sophisticated app bundle configuration for associated file types, identifier, version etc.
Graphical interface for bundling support files with script
Built-in script editor, or linking with external editor of choice
Set own application icon or select from presets
Command line tool for automating script application bundling
"Profiles" which can be used to save configurations

Official documentation for Python on Mac OS X

Needle is a tool for testing your CSS with Selenium and nose.

It checks that CSS renders correctly by taking screenshots of portions of a website and comparing them against known good screenshots. It also provides tools for testing calculated CSS values and the position of HTML elements.

# py-chrome-bookmarks

Simple Python scripts to convert [Google Chrome](http://www.google.com/chrome)’s bookmarks and history to the [standard HTML-ish bookmarks file format](http://msdn.microsoft.com/en-us/library/aa753582%28v=vs.85%29.aspx). Scripts written by Benjamin Esham <bdesham@gmail.com>.

The functionality to do this for bookmarks is already built into Chrome (select Bookmarks → Bookmarks Manager, then click “Organize” and select “Export Bookmarks…”). I wrote this script to be able to perform this conversion in a cron script.

## Usage

### Bookmarks script

From the command line, do

python py-chrome-bookmarks.py .../path/to/Chrome/Bookmarks output.html

The script will ignore URLs that start with “javascript:”.

### History script

From the command line, do

python py-chrome-history.py .../path/to/Chrome/History output.html

**Note:** it’s probably necessary to quit Chrome before running this so that the history database isn’t locked. Alternately, make a copy of Chrome’s History file and run the script on that.

The script will ignore history entries with empty titles.

#!/usr/bin/python

import json
import datetime
import cgi
import os
import sys

user = os.getlogin()
input_filename = "/Users/%s/Library/Application Support/Google/Chrome/Default/Bookmarks" % user
# modify if necessary
output_filename = "/Users/%s/Documents/chrome-bookmarks.html" % user

input_file = open(input_filename)
bookmark_data = json.load(input_file)
output_file = open(output_filename, 'w', 0)
sys.stdout = output_file

def print_bookmarks(bookmarks):
   for entry in bookmarks:
      if entry['type'] == 'folder':
         if not len(entry['children']) == 0:
            print ('<DT><H3 FOLDED ADD_DATE="%s">%s</H3>' % (entry['date_added'], entry['name'])).encode('utf-8')
            next_folder = entry['children']
            print '<DL><p>'
            print_bookmarks(next_folder)
            print '</DL><p>'
      else:
         print ('<DT><A DATE_ADDED="%s" HREF="%s">%s</A>' % (entry['date_added'], cgi.escape(entry['url']), entry['name'])).encode('utf-8')


print '<!DOCTYPE NETSCAPE-Bookmark-file-1>\n<Title>Bookmarks</Title>\n<H1>Bookmarks</H1><DL>\n'

roots = bookmark_data['roots']

for entry in roots:
   print ('<DT><H3 FOLDED ADD_DATE="%s">%s</H3>' % (roots[entry]['date_added'], entry)).encode('utf-8')
   print_bookmarks(roots[entry]['children'])
   print "</DL><p>"
   
print "</DL>"

%B %d, %Y

Cinderella is a fully managed development environment for open source hacking on Mac OSX. It's powered by rvm, homebrew and chef. You only need Xcode to get started.

Cinderella builds everything up in ~/Developer. It won't stomp on any of your current installations so you don't have to commit your entire machine immediately. It's simple to rollback if you really want to.

I'm trying to amass a list of programming books with opensource licenses, like Creative Commons, GPL, etc. The books can be about a particular programming language or about computers in general.

OK, I’ll confess upfront I’ve not tried this one yet although I hope to tomorrow. I got this info from Ars.

The basic problem is when you have a movie you’ve ripped but you want to watch it both on your Apple TV and your iPhone. Now ideally you’d want two different sizes. The iPhone doesn’t need the resolution you’d rip for the Apple TV plus you want the size to be as small as possible. Now if you’ve ever bought a show from iTunes you’ll note that Apple gives you both hi-res and low res versions. How do they do this? Well it turns out it’s just a tag on the MPEG-4 file. And there is a shell script to set those tags so you can do it with your own files.

First download and install mp4v2. Yes, you have to compile it. However if you have MacPorts you can also install it that way.

the trickiest part of cleaning ones iTunes library is looking up missing album information. There are a few applications that do this by mathematically analyzing the song’s music itself against a library. The problem is that this isn’t that accurate. There was an old app that did this (no longer available) called iEatzBrains or something like that. Unfortunately I made the mistake of running it against a bunch of tracks after testing it. Only then did I find that it’s accuracy was about 90%. So to be honest I am still finding tracks that had their artist and album names “revised” because of that. (I’d say that event caused about half of my library errors)

So I’m really leery of that sort of thing. However trying to find a script that could give a guess at album names was surprisingly hard. There weren’t any. There was a database of album information called discogs but it really didn’t let you search by track info. It was all oriented around correct album information. (Although it is useful for other metadata – as we’ll see in the future)

I did some pretty elaborate Google searching and remarkably this isn’t a problem that folks have solved. (Or at least they haven’t put solutions up online) Fortunately I found a solution from an unusual source: Apple.

Apple actually has a web API for searching the iTunes store. The documentation (oddly labeled “confidential” but publicly available) is quite helpful. What I did was to do an artist search returning just track information. Effectively getting a list of all songs by that artist. The returned data is even in JSON format rather than XML! This means I can make all the returned data into a Python dict with the simplejson module and a single call.

All I do then is just compare the selected song in iTunes (using Appscript) with the songs in the iTunes store. Now I could make this part a bit more intelligent by normalizing unicode characters and perhaps removing padding words like “a” or “the.” Thus far I’ve not had any trouble so I’ve not bothered making it “brighter.”

To avoid the problem of entering album information that is wrong and then never being able to find the changed files I put pretty extensive comments in each track. If the song could be from more than one album I put the other alternative albums in. Even if (as is typical) there’s only one album the track is on I put a note in the comments.

What’s surprising is how short the code is. (Take out all the comments in the code and there really isn’t much to it) My previous attempt using discog’s XML was at least 3 – 4 times as long and not nearly as accurate.

There are some amazing Python programmers out there who can do some pretty impressive and abstract stuff. However most of what I use Python for is as a quick and dirty scripting system. I’m rarely writing full elegant programs. Rather I just want to get something done and done quickly. Most of the time I’m not looking for every potential boundary problem. I’ll be able to tell if it doesn’t do what I want!

One thing I sometimes want to do it fix names in iTunes. That’s because I have some files where the name is mangled. (It’s all lower case, it has the track number prepended, or the like) While it’s possible to write a general case solution to this problem it’s much better to just have a script framework and modify a few lines for the particular problem.

Here’s how I do it using Appscript.

I wanted to start with the finding of duplicates. Now you could use a program like fdupes or dupes.py (discussed a few weeks ago). Those aren’t ideal. In the past I’ve used Doug’s Applescripts. He has several useful scripts. One is Remove Dead Tracks which gets those entries where the file is long gone. (I don’t know how that happens – it’s usually stuff I don’t want anyway) His other famous script is Corral iTunes Dupes which I’ve used in the past to find duplicate tracks.

What this script does is look up the standardized spelling for artists. As ever I work on a playlist you create. In this case the playlist “__Fix Arists” (without the quotes) I’d suggest adding to this list manually rather than trying to automate it. Both for safety (because it’s easy to undo if you just have a single artist in there) but also it just keeps things simple. If you need you could easily add in a routine to work on a selection. I’ll put that in the script but comment it out.

This is the second in a series of posts on cleaning up my library in iTunes. Last time we discussed using Dupin to find duplicates. This week I want to focus on the problem of capitalization of song names. If you are like me you have a few songs in your library where the case is screwed up. (Sometimes even the database of CD song names has some mistakes – most of my bad titles were from ripped CDs) What I want to do is use TitleCase (discussed last week) to check to see if the titles are correct. Because some titles are supposed to be weird rather than just automatically fixing things I just add the files to a special playlist. To keep my “fix it” playlists separate from my regular ones I always prepend them with two underscore characters. (I usually then delete them when I’m done — although you don’t have to)

TextMate is currently my editor of choice. For a long time I was a BBEdit fan but until recently it had really fallen behind. (I’ve heard the latest version is quite good though) Given a choice (at the time) between using XCode, paying to upgrade my copy of BBEdit, or buying TextMate, I went with TextMate. Now TextMate does have a few flaws. (The lack of real Applescript support for the text window and a slit pane window drive me bonkers) The author has been doing a rewrite but no one know when it’ll be out.

Where TextMate shines though is its thorough integration with the shell and its ability to call scripts, such as Applescript. (See, for instance, http://macromates.com/screencast/applescript-1.mov">this screencast showing how to run Applescripts from within TextMate instead of Script Editor.) It also has a lot of advanced editor features such as code completion, keyword tagging, and so forth. There’s a great tutorial online for using TextMate I’d suggest you check out. It’s very powerful and once you get used to it you’ll never want to leave it.

To start let me give a great way to keep infrequently used shell scripts. If you are like me you have some shell commands you run only occasionally. For instance when I am trying to debug a network I often use nmap (installed via MacPorts). However because I do it so infrequently I have to remember the arguments to the command and then how I mapped the NAT. (Some networks use 10.0.0.x while others use 192.168.1.x and others 192.168.0.x) So I put the following as a service in Automator.

The best way to really leverage Applescript is to turn on GUI scripting and enable the Script menu in the menu bar. Go to Applications > Applescript > Applescript Utility.app. This is a nice little application Apple supplies that will turn these on. I’ll not talk about GUI scripting but it lets you control things like menus even if an application is not written to support scripting. So it’s nice for certain tight spots where an application has either no or poor Applescript support.

The Applescript menu creates a little menu item listing scripts for that application. Even though I’m not a fan of Applescript the fact is that there are a lot of great little utilities. Running them from the Applescript menu is much, much faster than running a separate application (say created from Apple’s Script Editor).

I prefer to script, as much as possible, in Python + Appscript. However sometimes it’s just more convenient to do it in Applescript. (Actually, some of the shorter Python scripts I do would probably be better done in Applescript – but I like doing them in Python so I can quickly extend them when the need arises.) What would be nice is to be able to have shell scripts that execute Applescript the way you can have them execute Python, bash, Ruby or so forth.

First download both ASTranslate and ASDictionary from the Appscript tool page. ASDictionary adds a help feature to Appscript. ASTranslate translates Applescript into Python. Both are essential if you want to script.

Next install iPython on your computer. Ignore most of the pages on iPython installation Google gives you. With recent versions of OSX Python comes with easy_install. Just type easy_install ipython at the command line and everything should get installed. If your system doesn’t have easy_install then install setuptools which includes it. (As an aside, if you’ve jailbroken your iPhone then iPython makes a wonderful calculator from the Cydia terminal)

I have just installed Snow Leopard and came upon some excellent sites which have inspired me to learn the programming language Python

The first site is Clark’s Tech Blog

The articles listed below are excellent examples of using Python to manage your itunes library, well worth the read.

* Fix iTunes Names
* Cleaning iTunes Pt. 1
* Cleaning iTunes Pt. 2: Title Case
* Cleaning iTunes Pt. 3: Album Names

#!/bin/sh
IFS=$'\n'
for line in `< $1`; do
youtube-dl -b -t $line
done

git clone git://github.com/acangiano/pygments-textmate-bundle.git "Pygments.tmbundle"

Review Board is a powerful web-based code review tool that offers developers an easy way to handle code reviews. It scales well from small projects to large companies and offers a variety of tools to take much of the stress and time out of the code review process.