Simple Full Text Search for App Engine

Revised 2009-07-12 to include addition of stemming, multi-word exact matching, multiple search entities, and stowing of designated parent titles in index entity keys.

The basic insight behind Levenshtein automata is that it's possible to construct a Finite state automaton that recognizes exactly the set of strings within a given Levenshtein distance of a target word. We can then feed in any word, and the automaton will accept or reject it based on whether the Levenshtein distance to the target word is at most the distance specified when we constructed the automaton. Further, due to the nature of FSAs, it will do so in O(n) time with the length of the string being tested. Compare this to the standard Dynamic Programming Levenshtein algorithm, which takes O(mn) time, where m and n are the lengths of the two input words! It's thus immediately apparrent that Levenshtein automaton provide, at a minimum, a faster way for us to check many words against a single target word and maximum distance - not a bad improvement to start with!

Search Index compression

The basic strategy is to take a collection of search terms, automatically query Google for the number of results associated with each of these search terms (displayed by Google when doing a web search), and graph these numbers over time with Gnuplot.