Jurafsky and Manning's online NLP course.

Clojure wrapper for Stanford CoreNLP tools.

Clojure library interface to OpenNLP.

The video “How Natural Language Processing is Changing Research” provides a more extended look at WordSeer’s usefulness for analyzing slave narratives, but its purpose is also to underscore how such a tool can benefit humanities scholars. In this video the discussion veers toward presenting reading as a chore from which humanities scholars seek relief. On that note, a student in Dr. Michael Ullyot’s undergraduate ENG 203 course, “Hamlet in the Humanities Lab” at the University of Calgary offers some pertinent comments. In her penultimate blog post for the course, Stephanie Vandework devotes a section to “The Pros and Cons of Exploratory Analysis” and examines more closely the claims in the WordSeer Shakespeare demo, finding some to suffer from overgeneralization. (For a view of the course from the instructor’s perspective, see Dr. Ullyot’s presentation, Teaching Hamlet in the Humanities Lab, for the Renaissance Society of America conference this past March 2012.)

NEH DH Lightning Round on Wordseer.

We present a probabilistic approach for learning to interpret temporal phrases given only a corpus of utterances and the times they reference. While most approaches to the task have used regular expressions and similar linear pattern interpretation rules, the possibility of phrasal embedding and modification in time expressions motivates our use of a compositional grammar of time expressions. This grammar is used to construct a latent parse which evaluates to the time the phrase would represent, as a logical parse might evaluate to a concrete entity. In this way, we can employ a loosely supervised EM-style bootstrapping approach to learn these latent parses while capturing both syntactic uncertainty and pragmatic ambiguity in a probabilistic framework. We achieve an accuracy of 72% on an adapted TempEval-2 task – comparable to state of the art systems.

Natural Language Understanding requires a large amount of background "common sense" knowledge about the situation under discussion. In many respects, using this knowledge is at the core of reasoning and acting in traditional Artificial Intelligence. When reading an article about a criminal conviction, the writer assumes the reader knows about trials, juries, and criminal activity. The Narrative Chain project aims to learn this knowledge by processing large amounts of text and learning which events tend to occur together. We are studying not just what can be learned, but also the best representation for this knowledge (graph, linear chain, frame?).

This project also includes research into ordering events in time. For instance, did the conviction or the sentencing happen first? We use modern machine learning techniques to find linguistic features that indicate this semantic ordering relation.

An example of a learned narrative event chain, with arrows indicating temporal ordering, is shown on the right. The bold words are the events, and the subj/obj terms indicate how the common actor in this narrative is involved in the event (the subject or object of the verb).

The amount of literary material available on-line keeps growing rapidly. Not only are there machine-readable texts in libraries, collections and e-book stores, but there is also more and more “live” literature – e-zines, blogs, self-published e-books and so on. There is a need for tools to help users navigate, visualize and appreciate high volume of available literature.

Literary texts are quite different from technical and formal documents, which have been the focus of NLP research thus far. Most forms of statistical language processing rely on lexical information in one way or another. In literature, the primary mode is narrative rather than exposition. Stories may be cognitively easier to read than certain expository genres, such as scientific documents, but it is a challenging form of discourse for NLP tools and methods. For instance, literary prose lacks overt lexical clues and structural markers typically leveraged in the processing of more structured genres. Also, even conventional literary texts exhibit far less unity of time, space and topic than most formal discourse. Learning to handle these challenges in literary data may help move past heavy reliance on surface clues in general.

Literature also differs from other genres because of the needs of its typical audience. For instance, reading, searching or browsing literature online is a different task than searching for the latest news on a particular topic. Search criteria would be rather abstract: not a keyword, but a literary style, similarity to another work, point of view and so on. When looking for a summary or a digest, a reader may prefer to know or visualize a text's broad characteristics than facts which summarize the plot.

We invite papers that touch upon these areas, but also welcome other ideas which promote the processing of literary narrative or related forms of discourse.

Daniel Marcu's discourse structure reading group at ISI.

The obvious question is how one manages to distinguish among summary, analysis, argument, and interpretation. E.g.

With the aid of a rag tag crew of adventurers, a young man rescues a princess from an evil empire and discovers his destiny to become a member of a dying order of knights.

A young man helps a rebel leader escape from an imperial prison and participates in an pitched battle to save the rebels' military base.

I assume you recognize the story, and I think most people would say the first summary is more accurate. Why? The second one is certainly not inaccurate. It simply downplays the "hero's journey" aspect and portrays the film as depicting a political and collective activity.

What are the N most similar words to X, according to WordNet?

This data seeks to answer that question, where similarity is based on
measures from WordNet::Similarity. http://wn-similarity.sourceforge.net

Apache Stanbol (currently in incubation) is an open source modular software stack and reusable set of components for semantic content management.

Apache Stanbol components are meant to be accessed over RESTful interfaces to provide semantic services for content management. Thus, one application is to extend traditional content management systems with (internal or external) semantic services.

Chris Manning's tutorial at Digital Humanities 2011 at Stanford.

We describe our contribution to the SemEval task on Frame-Semantic Structure Extraction. Unlike most previous systems described in literature, ours is based on dependency syntax. We also describe a fully automatic method to add words to the FrameNet lexical database, which gives an improvement in the recall of frame detection.

Our annotation tool follows the tag-a-little, train-a-little paradigm, in which an automatic system based on the already-annotated data is trained as you go to pre-annotate the data for a user to correct.

Includes an implementation of PLDA.

Partially Labeled Dirchlet Allocation (PLDA) [paper] is a topic model that extends and generalizes both LDA and Labeled LDA. The model is analogous to Labeled LDA except that it allows more than one latent topic per label and a set of background labels. Learning and inference in the model is much like the example above for Labeled LDA, but you must additionally specify the number of topics associated with each label.

The Natural Language Software Registry (NLSR) is a concise summary of the capabilities and sources of a large amount of natural language processing (NLP) software available to the NLP community. It comprises academic, commercial and proprietary software with specifications and terms on which it can be acquired clearly indicated.

The goal of the study is to examine work that has something to offer toward the construction of a computable model of text understanding. It focuses on those aspects of meaning that are conveyed only by groups of connected sentences—texts. Additionally, only work that attempts to deal with the semantics or understanding of texts, as opposed to statistical or syntactic analysis, is considered.

The iDASH NLP Ecosystem is a place to share and access tools, data, and educational resources for developing and applying NLP to clinical text. 2011-2012 talks on temporal reasoning.

These n-grams are based on the largest publicly-available, genre-balanced corpus of English -- the 425 million word Corpus of Contemporary American English (COCA). With this n-grams data (2, 3, 4, 5-word sequences, with their frequency), you can carry out powerful queries offline -- without needing to access the corpus via the web interface.

Conditional random fields (CRFs) are a probabilistic framework for labeling and segmenting structured data, such as sequences, trees and lattices. The underlying idea is that of defining a conditional probability distribution over label sequences given a particular observation sequence, rather than a joint distribution over both label and observation sequences. The primary advantage of CRFs over hidden Markov models is their conditional nature, resulting in the relaxation of the independence assumptions required by HMMs in order to ensure tractable inference. Additionally, CRFs avoid the label bias problem, a weakness exhibited by maximum entropy Markov models (MEMMs) and other conditional Markov models based on directed graphical models. CRFs outperform both MEMMs and HMMs on a number of real-world tasks in many fields, including bioinformatics, computational linguistics and speech recognition.

Sentence tokenizer written in python. Includes proper tokenization and models for very high accuracy sentence boundary detection (English only for now). The models are trained from Wall Street Journal news combined with the Brown Corpus which is intended to be widely representative of written English. Error rates on test news data are near 0.25%.

Visualizing and analyzing social networks is a challenging problem that has been receiving growing attention. An important first step, before analysis can begin, is ensuring that the data is accurate. A common data quality problem is that the data may inadvertently contain several distinct references to the same underlying entity; the process of reconciling these references is called entity resolution. D-Dupe is an interactive tool that combines data mining algorithms for entity resolution with a task-specific network visualization. Users cope with complexity of cleaning large networks by focusing on a small subnetwork containing a potential duplicate pair. The subnetwork highlights relationships in the social network, making the common relationships easy to visually identify. D-Dupe users resolve ambiguities either by merging nodes or by marking them distinct. The entity resolution process is iterative: as pairs of nodes are resolved, additional duplicates may be revealed; therefore, resolution decisions are often chained together. We give examples of how users can flexibly apply sequences of actions to produce a high quality entity resolution result.

Great, clear example of text mining using Wordseer.

Event coreference is an important and complicated task in cascaded event template extraction and other natural language processing tasks. Despite its importance, it was merely discussed in previous studies. In this paper, we present a globally optimized coreference resolution system dedicated to various sophisticated event coreference phenomena. Seven resolvers for both event and object coreference cases are utilized, which include three new resolvers for event coreference resolution. Three enhancements are further proposed at both mention pair detection and chain formation levels. First, the object coreference resolvers are used to effectively reduce the false positive cases for event coreference. Second, A revised instance selection scheme is proposed to improve link level mention-pair model performances. Last but not least, an efficient and globally optimized graph partitioning model is employed for coreference chain formation using spectral partitioning which allows the incorporation of pronoun coreference information. The three techniques contribute to a significant improvement of 8.54% in B 3 F-score for event coreference resolution on OntoNotes 2.0 corpus.

The overall goal is to discuss some issues concerning the dependencies at the discourse level and at the sentence level. However, first I will briefly describe the Penn Discourse Treebank (PDTB)*, a corpus in which we annotate the discourse connectives (explicit and implicit) and their arguments together with "attributions" of the arguments and the relations denoted by the connectives, and also the senses of the connectives. I will then focus on the complexity of dependencies in terms of (a) the elements that bear the dependency relations, (b) graph theoretic properties of these dependencies such as nested and crossed dependencies, dependencies with shared arguments, and (c) attributions and their relationship to the dependencies, among others. I will compare these dependencies with those at the sentence level and discuss some issues that relate to the transition from the sentence level to the level of "immediate discourse" and propose some conjectures.

This is a straight port from the CLD (Compact Language Detector) library embedded in Google's Chromium browser. The library detects the language from provided UTF8 text (plain text or HTML). It's implemented in C++, with very basic Python bindings.

Dan Jurafsky, Syntactic Variations Versus Semantic Roles, Some Typical Semantic Roles, Two Solutions To The Difficulty Of Defining Semantic Roles, PropBank, FrameNet, Information Extraction Versus Semantic Role Labeling, Evaluation Measures, Parsing Algorithm, Combining Identification And Classification Models, Summary

The description of events in biomedical literature often follows discourse patterns.
For example, authors may firstly mention the transcription of a
gene, and then go on to describe how this transcription is regulated by another
gene. Capturing such patterns can be beneficial when we want to extract event
mentions from literature. For instance, detecting the mention of a transcription of
gene A gives us a hint to actively look for mentions of regulations involving A.
With this hint we could find such mentions even if they follow unseen lexical or
syntactic patterns. To exploit such hints we need to perform event extraction in a
cross sentence manner.

It is shown that imperatively defined factor graphs (IDF) are an intuitive way to
build Markov Networks that model inter-dependencies between mentions of events
within sentences, and across sentence-boundaries. Small pieces of procedural code
define the graph structure, feature functions and hooks for efficient inference.
Empirically, this leads to an efficient cross-sentence event extractor with very
competitive results on the BioNLP shared task. One of our inter-event features
shows an impact of 1:94 points in F1 for the class of regulation events.

On this page, we break our competition down into academic toolkits and industrial toolkits. We only consider software that is available for linguistic processing, not companies that rely on linguistic processing in an application but do not sell that technology.

Semantic Role Labeler is a machine-learning based tool that analyzes for a shallow semantic information of a given sentence. The tool is capable of outputing verb-argument structure following the notation defined by the Propbank project.

The tools provide a pipeline of modules that carry out lemmatization, part-of-speech tagging, dependency parsing, and semantic role labeling of a sentence. The system’s two main components draw on improved versions of a state-of-the-art dependency parser (Bohnet, 2010) and semantic role labeler (Björkelund et al.,2009) developed independently by the authors. The tools are language independent, provide a very high accuracy and are fast. The dependency parser had the top score for German and English dependency parsing in the CoNLL shared task 2009.

In recent years, researchers in several communities involved in aspects of the web have begun to realise the potential benefits of assigning an important role to events in the representation and organisation of knowledge and media. While a good deal of relevant research has been done in the semantic web community (for example on the modeling of events), a lot of complementary research has been done in other communities, such as multimedia processing and information retrieval. The goal of this workshop is to advance research on this general topic within the semantic web community, by both building on existing semantic web work and integrating results and methods from other areas, with a particular focus on issues that are central to the semantic web.

In this paper we present a novel, customizable IE paradigm that takes advantage of predicate-argument structures. We also introduce a new way of automatically identifying predicate argument structures, which is central to our IE paradigm. It is based on: (1) an extended set of features; and (2) inductive decision tree learning. The experimental results prove our claim that accurate predicate-argument structures enable high quality IE results.

Demo of mate-tools semantic role labeler.

VerbNet (VN) (Kipper-Schuler 2006) is the largest on-line verb lexicon currently available for English. It is a hierarchical domain-independent, broad-coverage verb lexicon with mappings to other lexical resources such as WordNet (Miller, 1990; Fellbaum, 1998), Xtag (XTAG Research Group, 2001), and FrameNet (Baker et al., 1998). VerbNet is organized into verb classes extending Levin (1993) classes through refinement and addition of subclasses to achieve syntactic and semantic coherence among members of a class. Each verb class in VN is completely described by thematic roles, selectional restrictions on the arguments, and frames consisting of a syntactic description and semantic predicates with a temporal function, in a manner similar to the event decomposition of Moens and Steedman (1988).

Proposition Bank I was produced by Linguistic Data Consortium (LDC) catalog number LDC2004T14 and ISBN 1-58563-304-6.

This is a semantic annotation of the Wall Street Journal section of Treebank-2. More specifically, each verb occurring in the Treebank has been treated as a semantic predicate and the surrounding text has been annotated for arguments and adjuncts of the predicate. The verbs have also been tagged with coarse grained senses and with inflectional information. This work was done in the Computer and Information Sciences Department at the University of Pennsylvania.

The original PropBank project, funded by ACE, created a corpus of text annotated with information about basic semantic propositions. Predicate-argument relations were added to the syntactic trees of the Penn Treebank. This resource is now available via LDC.

SemLink is a project whose aim is to link together different lexical resources via a set of mappings. These mappings will make it possible to combine the different information provided by these different lexical resources for tasks such as inferencing. We also plan to use the mappings to aid in semi-automatic extension of each resources coverage, to increase the overall overlap in coverage. Currently, we are creating mappings between the following resources:

PropBank: A corpus of one million words of English text, annotated with argument role labels for verbs; and a lexicon defining those argument roles on a per-verb basis.
VerbNet: A lexicon that groups verbs based on their semantic/syntactic linking behavior.
FrameNet: A lexicon based on frame semantics.
WordNet: A lexicon that describes semantic relationships (such as synonymy and hyperonymy) between individual words.

SEMAFOR: Semantic Analysis of Frame Representations is a tool for automatic analysis of the frame-semantic structure of English text.

ScalaNLP is a collection of libraries for Natural Language Processing, Machine Learning, and Statistics.

This chapter studies the use of textual features based on systemic functional linguistics, for genre-based text categorization. We describe feature sets that represent different types of conjunctions and modal assessment, which together can partially indicate how different genres structure text and may prefer certain classes of attitudes towards propositions in the text. This enables analysis of large-scale rhetorical differences between genres by examining which features are important for classification. The specific domain we studied comprises scientific articles in historical and experimental sciences (paleontology and physical chemistry, respectively). We applied the SMO learning algorithm, which with our feature set achieved over 83% accuracy for classifying articles according to field, though no field-specific terms were used as features. The most highly-weighted features for each were consistent with hypothesized methodological differences between historical and experimental sciences, thus lending empirical evidence to the recent philosophical claim of multiple scientific methods.

FOX is a framework that integrates the Linked Data Cloud and makes uses of the diversity of NLP algorithms to extract RDF triples of high accuracy out of NL. In its current version, it integrates and merges the results of Named Entity Recognition, Keyword Extraction and Relation Extraction tools.

The course will synthesize recent research in linguistics, computer science, and natural language processing with the aim of introducing students to theoretical and computational models of language. The course will familiarize students with a wide range of linguistic phenomena with the aim of appreciating the complexity, but also the systematic behaviour of natural languages like English, the pervasiveness of ambiguity, and how this presents challenges in natural language processing. In addition, the course introduce the most important algorithms and data structures that are commonly used to solve many NLP problems.

This course is a graduate level introduction to automatic discourse processing. The emphasis will be on methods and models that have applicability to natural language and speech processing.

The class will cover the following topics: discourse structure, models of coherence and cohesion, plan recognition algorithms, and text segmentation. We will study symbolic as well as machine learning methods for discourse analysis. We will also discuss the use of these methods in a variety of applications ranging from dialogue systems to automatic essay writing.

Proceedings of the 5th ACL-HLT Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities.

The LaTeCH workshop series aims to provide a forum for researchers who are working on developing novel information technology for improved information access to data from the Humanities, Social Sciences, and Cultural Heritage.

Recent developments in the Humanities, Social Sciences, and Cultural Heritage draw an increasing interest from researchers in NLP in developing methods for data cleaning, semantic annotation, intelligent querying, linking, discovery and visualisation of interesting trends. Language technology has an important role to play in these processes, even for collections which are primarily non-textual, since text is the pervasive medium used for metadata. These fairly novel domains of application entail new challenges to NLP research, such as noisy text (e.g., due to OCR problems), non-standard, or archaic language varieties (e.g., historic language, dialects, mixed use of languages, ellipsis, transcription errors), the necessity to link data of diverse formats (e.g., text, database, video, speech) and languages, and the lack of available resources, such as dictionaries. Furthermore, often neither annotated domain data is available, nor the required funds to manually create it, thus forcing researchers to investigate (semi-) automatic resource development and domain adaptation approaches involving the least possible manual effort.

Standard algorithms for template-based in- formation extraction (IE) require predefined template schemas, and often labeled data, to learn to extract their slot fillers (e.g., an embassy is the Target of a Bombing tem- plate). This paper describes an approach to template-based IE that removes this require- ment and performs extraction without know- ing the template structure in advance. Our al- gorithm instead learns the template structure automatically from raw text, inducing tem- plate schemas as sets of linked events (e.g., bombings include detonate, set off, and de- stroy events) associated with semantic roles. We also solve the standard IE task, using the induced syntactic patterns to extract role fillers from specific documents. We evaluate on the MUC-4 terrorism dataset and show that we in- duce template structure very similar to hand- created gold structure, and we extract role fillers with an F1 score of .40, approaching the performance of algorithms that require full knowledge of the templates.

Nested event structures are a common occur- rence in both open domain and domain spe- cific extraction tasks, e.g., a “crime” event can cause a “investigation” event, which can lead to an “arrest” event. However, most cur- rent approaches address event extraction with highly local models that extract each event and argument independently. We propose a simple approach for the extraction of such structures by taking the tree of event-argument relations and using it directly as the representation in a reranking dependency parser. This provides a simple framework that captures global prop- erties of both nested and flat event structures. We explore a rich feature space that models both the events to be parsed and context from the original supporting text. Our approach ob- tains competitive results in the extraction of biomedical events from the BioNLP’09 shared task with a F1 score of 53.5% in development and 48.6% in testing.

Yahoo!'s topic modelling framework using Latent Dirichlet Allocation.

KEA is an algorithm for extracting keyphrases from text documents. It can be either used for free indexing or for indexing with a controlled vocabulary.

Maui automatically identifies main topics in text documents. Depending on the task, topics are tags, keywords, keyphrases, vocabulary terms, descriptors, index terms or titles of Wikipedia articles.

Maui performs the following tasks:

term assignment with a controlled vocabulary (or thesaurus)
subject indexing
topic indexing with terms from Wikipedia
keyphrase extraction
terminology extraction
automatic tagging
It can also be used for terminology extraction and semi-automatic topic indexing.

Wikipedia Miner is a toolkit for navigating and making use of the structure and content of Wikipedia. It aims to make it easy for you to integrate Wikipedia's knowledge into your own applications, by:

providing simplified, object-oriented access to Wikipedia's structure and content.
measuring how terms and concepts in Wikipedia are connected to each other.
detecting and disambiguating Wikipedia topics when they are mentioned in documents.

Alphabetical list of part-of-speech tags used in the Penn Treebank Project.

Example of using Tregex and Tsurgeon.

TiMBL is an open source software package implementing several memory-based learning algorithms, among which IB1-IG, an implementation of k-nearest neighbor classification with feature weighting suitable for symbolic feature spaces, and IGTree, a decision-tree approximation of IB1-IG. All implemented algorithms have in common that they store some representation of the training set explicitly in memory. During testing, new cases are classified by extrapolation from the most similar stored cases.

For the past decade, TiMBL has been mostly used in natural language processing as a machine learning classifier component, but its use extends to virtually any supervised machine learning domain. Due to its particular decision-tree-based implementation, TiMBL is in many cases far more efficient in classification than a standard k-nearest neighbor algorithm would be.

A collection of the best open data sets and open-source tools for data science, wrapped in an easy-to-use REST/JSON API with command line, Python and Javascript interfaces. Available as a self-contained VM or EC2 AMI that you can deploy yourself.

Lippmannian device is named after Lippmann, and provides a coarse means of showing actor partisanship.

This package contains the different data structures used by JavaNLP throughout the years for dealing with linguistic objects in general, of which words are the most generally used.

A python interface to the Stanford Parser. It uses JPype to create a Java virtual machine, instantiate the parser, and call methods on it. Most of the code is focused on getting the Stanford Dependencies, but it's easy to add API to call any method on the parser.

...digital media and huge databases have enormous potential for supporting, preserving, and making available for study the kinds of underground knowledges and cultural productions outside the sphere of mainstream print that you're concerned about. This is the insurgent potential of the Internet and digital media--they can bypass established methods of fixation and legitimation of cultural products. But in academia these are subjects of interest to humanists--and sociologists and anthropologists. By contrast, when true disciplinary outsiders like Jean-Baptiste Michel and his team enter the arena of cultural history and cultural studies from the side of science and engineering, they must be looking to legitimate themselves by proving that their approach "works" for subjects that they imagine will be widely recognized as significant.

...the lesson that I would then focus on with my students is that what they are looking at in such a graph is nothing more or less than the frequency with which a word is used in book (and only books) published over the centuries. While such frequencies do reflect something, it is not clear from one graph just what that something is. So instead of an answer, a graph like this one is a doorway that leads to a room filled with questions, each of which must be answered by the historian before he or she knows something worth knowing.

There's a certain Words For Snowism in the online Google Books Ngrams tool, the suggestion that the more frequently a word is used, the more important it is in a collective unconscious of which the Google Books data set serves as a convenient index. This importance is not the same thing as significance, in the sense of significant digits or statistical significance; it's not the difference that makes a difference, but rather a psychologized importance--attachment, cathexis. Which is really kind of garbage.

Stanford CoreNLP provides a set of natural language analysis tools which can take raw English language text input and give the base forms of words, their parts of speech, whether they are names of companies, people, etc., normalize dates, times, and numeric quantities, and mark up the structure of sentences in terms of phrases and word dependencies, and indicate which noun phrases refer to the same entities. It provides the foundational building blocks for higher level text understanding applications.

Stanford CoreNLP integrates all our NLP tools for the English language, including the part-of-speech (POS) tagger, the named entity recognizer (NER), the parser, and the coreference resolution system. The goal of this project is to enable people to quickly and painlessly get complete linguistic annotations of natural language texts. It is designed to be highly flexible and extensible, i.e., with a single option you can change which tools should be enabled and which should be disabled.

tm (shorthand for Text Mining Infrastructure in R) provides a framework for text mining applications within R.

The tm package offers functionality for managing text documents, abstracts the process of document manipulation and eases the usage of heterogeneous text formats in R. The package has integrated database backend support to minimize memory demands. An advanced meta data management is implemented for collections of text documents to alleviate the usage of large and with meta data enriched document sets.

The GEOLocate project is an effort to develop software and services for translating textual locality descriptions associated with biodiversity collections data into geographic coordinates.

MALLET is a Java-based package for statistical natural language processing, document classification, clustering, topic modeling, information extraction, and other machine learning applications to text.

Giving a broad perspective of the field from numerous vantage points, Text Mining: Classification, Clustering, and Applications focuses on statistical methods for text mining and analysis. It examines methods to automatically cluster and classify text documents and applies these methods in a variety of areas, including adaptive information filtering, information distillation, and text search.

The book begins with chapters on the classification of documents into predefined categories. It presents state-of-the-art algorithms and their use in practice. The next chapters describe novel methods for clustering documents into groups that are not predefined. These methods seek to automatically determine topical structures that may exist in a document corpus. The book concludes by discussing various text mining applications that have significant implications for future research and industrial use.

The Text Processing API supports the following functionality:

Stemming & Lemmatization
Sentiment Analysis
Tagging and Chunk Extraction

We're writing a book about LingPipe. As it's written, we'll be putting up drafts here.

SiLCC is a cloud based service for parsing text and extracting relevant keywords.

Where data geeks ask and answer questions on machine learning, natural language processing, artificial intelligence, text analysis, information retrieval, search, data mining, statistical modeling, and data visualization!

BigSheets is an extension of the mashup paradigm that:
1. Integrates gigabytes, terabytes, or petabytes of unstructured data from web-based repositories
2. Collects a wide range of unstructured web data stemming from user-defined seed URLs
3. Extracts and Enriches that data using the unstructured information management architecture you choose (LanguageWare,OpenCalais, etc.)
4. Lets you Explore and Visualize this data in specific, user defined contexts. (such as ManyEyes)

In search of a good general-purpose method of pulling the content out of arbitrary web pages and leaving the boilerplate, advertising, navigation, etc. behind. See also http://bit.ly/4SFOIH

Maximum entropy (aka logistic regression) models are very popular, especially in natural language processing. The software here is an implementation of maximum likelihood and maximum a posterior optimization of the parameters of these models. The algorithms used are much more efficient than the iterative scaling techniques used in almost every other maxent package out there.

This package determines important terms within a given piece of content. It uses linguistic tools such as Parts-Of-Speech (POS) and some simple statistical analysis to determine the terms and their strength.

LingPipe is a suite of Java libraries for the linguistic analysis of human language.

Named entity chunker for NLTK.

Tools to identify collocations --- words that often appear consecutively --- within corpora. They may also be used to find other associations between word occurrences.