"Searching for relevant content in a massive amount of multimedia information is facilitated by accurately annotating each image, video, or song with a large number of relevant semantic keywords, or tags. We introduce game-powered machine learning, an integrated approach to annotating multimedia content that combines the effectiveness of human computation, through online games, with the scalability of machine learning. We investigate this framework for labeling music. First, a socially-oriented music annotation game called Herd It collects reliable music annotations based on the “wisdom of the crowds.” Second, these annotated examples are used to train a supervised machine learning system. Third, the machine learning system actively directs the annotation games to collect new data that will most benefit future model iterations. Once trained, the system can automatically annotate a corpus of music much larger than what could be labeled using human computation alone. Automatically annotated songs can be retrieved based on their semantic relevance to text-based queries (e.g., “funky jazz with saxophone,” “spooky electronica,” etc.). Based on the results presented in this paper, we find that actively coupling annotation games with machine learning provides a reliable and scalable approach to making searchable massive amounts of multimedia data."

--- This is more than a bit of a stunt, but it points in an interesting direction.

"Over a period of more than 30 years, more than 100 mathematicians worked on a project to classify mathematical objects known as finite simple groups. The Classification, when officially declared completed in 1981, ranged between 300 and 500 articles and ran somewhere between 5,000 and 10,000 journal pages. Mathematicians have hailed the project as one of the greatest mathematical achievements of the 20th century, and it surpasses, both in scale and scope, any other mathematical proof of the 20th century. The history of the Classification points to the importance of face-to-face interaction and close teaching relationships in the production and transformation of theoretical knowledge. The techniques and methods that governed much of the work in finite simple group theory circulated via personal, often informal, communication, rather than in published proofs. Consequently, the printed proofs that would constitute the Classification Theorem functioned as a sort of shorthand for and formalization of proofs that had already been established during personal interactions among mathematicians. The proof of the Classification was at once both a material artifact and a crystallization of one community’s shared practices, values, histories, and expertise. However, beginning in the 1980s, the original proof of the Classification faced the threat of ‘uninvention’. The papers that constituted it could still be found scattered throughout the mathematical literature, but no one other than the dwindling community of group theorists would know how to find them or how to piece them together. Faced with this problem, finite group theorists resolved to produce a ‘second-generation proof’ to streamline and centralize the Classification. This project highlights that the proof and the community of finite simple groups theorists who produced it were co-constitutive–one formed and reformed by the other."

"Complex problems in science, business, and engineering typically require some tradeoff between exploitation of known solutions and exploration for novel ones, where, in many cases, information about known solutions can also disseminate among individual problem solvers through formal or informal networks. Prior research on complex problem solving by collectives has found the counterintuitive result that inefficient networks, meaning networks that disseminate information relatively slowly, can perform better than efficient networks for problems that require extended exploration. In this paper, we report on a series of 256 Web-based experiments in which groups of 16 individuals collectively solved a complex problem and shared information through different communication networks. As expected, we found that collective exploration improved average success over independent exploration because good solutions could diffuse through the network. In contrast to prior work, however, we found that efficient networks outperformed inefficient networks, even in a problem space with qualitative properties thought to favor inefficient networks. We explain this result in terms of individual-level explore-exploit decisions, which we find were influenced by the network structure as well as by strategic considerations and the relative payoff between maxima. We conclude by discussing implications for real-world problem solving and possible extensions."

"Boosting is an approach to machine learning based on the idea of creating a highly accurate predictor by combining many weak and inaccurate “rules of thumb.” A remarkably rich theory has evolved around boosting, with connections to a range of topics, including statistics, game theory, convex optimization, and information geometry. Boosting algorithms have also enjoyed practical success in such fields as biology, vision, and speech processing. At various times in its history, boosting has been perceived as mysterious, controversial, even paradoxical.

This book, written by the inventors of the method, brings together, organizes, simplifies, and substantially extends two decades of research on boosting, presenting both theory and applications in a way that is accessible to readers from diverse backgrounds while also providing an authoritative reference for advanced researchers. With its introductory treatment of all material and its inclusion of exercises in every chapter, the book is appropriate for course use as well.

The book begins with a general introduction to machine learning algorithms and their analysis; then explores the core theory of boosting, especially its ability to generalize; examines some of the myriad other theoretical viewpoints that help to explain and understand boosting; provides practical extensions of boosting for more complex learning problems; and finally presents a number of advanced theoretical topics. Numerous applications and practical illustrations are offered throughout."

"Machine Learning competitions such as the Netflix Prize have proven reasonably successful as a method of "crowdsourcing" prediction tasks. But these competitions have a number of weaknesses, particularly in the incentive structure they create for the participants. We propose a new approach, called a Crowdsourced Learning Mechanism, in which participants collaboratively "learn" a hypothesis for a given prediction task. The approach draws heavily from the concept of a prediction market, where traders bet on the likelihood of a future event. In our framework, the mechanism continues to publish the current hypothesis, and participants can modify this hypothesis by wagering on an update. The critical incentive property is that a participant will profit an amount that scales according to how much her update improves performance on a released test set."

"The process of developing a new technology through open discussion has been called collective invention. This paper documents two episodes of collective invention and proposes a general model based on search theory. The first episode deals with the development of mass production steel in the U.S. (1866-1885), and the second with early personal computers (1975- 1985). In both cases technical people openly discussed and sometimes shared technology they were developing. Both technologies advanced to the point that they supported substantial economic growth. Open source software development is partway through a similar process now.
The episodes have common features. The process begins with an invention or a change in legal restrictions. Hobbyists and startup firms experiment with practical methods of production and share their results through a social network. The members of the network form a new industry or change an existing one. The network then disappears if the new firms keep their research and development secret. A model of the search for innovations can describe this process if it is expanded to include independent hobbyists and consultants as well as profit-seeking firms."

"The aggregation of consistent individual judgments on logically interconnected propositions into a collective judgment on those propositions has recently drawn much attention. Seemingly reasonable aggregation procedures, such as propositionwise majority voting, cannot ensure an equally consistent collective conclusion. In this paper, we motivate that quite often, we do not only want to make a factually right decision, but also to correctly evaluate the reasons for that decision. In other words, we address the problem of tracking the truth. We set up a probabilistic model that generalizes the analysis of Bovens and Rabinowicz (2006) and use it to compare several aggregation procedures. Demanding some reasonable adequacy constraints, we demonstrate that a reasons- or premise-based aggregation procedure tracks the truth better than any other procedure. However, we also illuminate that such a procedure is not in all circumstances easy to implement, leaving actual decision-makers with a tradeoff problem."

"This paper argues that democracy can be seen as a way to channel “democratic reason,” or the collective political intelligence of the many. The paper hypothesizes that two main democratic mechanisms - the practice of inclusive deliberation (in its direct and indirect versions) and the institution of majority rule with universal suffrage - combine their epistemic properties to maximize the chances that the group pick the “better” political answer within a given context and a set of values. The paper further argues that under the conditions of a liberal society, characterized among other things by sufficient cognitive diversity, these two mechanisms give democracy an epistemic edge over versions of the rule of the few."

Sounds like it's more useful as source material than for any conclusions.

Heard the talk at ISIT 2011; it was very good.

"the Borat humor is taking people whose jobs are to behave a certain way under a familiar, professionalized script and then start acting like a weirdo. ... They all try to keep to their scripts while the person opposite of them acts like a buffoon,,,, instead of going “stop acting like a buffoon.” ... These right-wing videos take this and amplify a particularly interesting part of the service/professionalized economy. When so much of our economy is driven by professionals there is a lot of work done in making sure that there are layers of people between the consumer and the professional. ... You don’t want the expensive brain surgeon making sure you’ve filled out your address and contact information correctly or taking your temperature – that’s why there’s a secretary and a nurse in-between these steps at the hospital.What the right-wing videos do ... is present the front-line staff as the actual decision making professionals. ..."

Review: http://bactra.org/weblog/algae-2011-08.html#laughlin

Actually, I heard the talk about this and some even more impressive follow-up work, which Jadbabaie said will be in working paper form very soon.

"Just as neurons interconnect in networks that create structured thoughts beyond the ken of any individual neuron, so people spontaneously organize themselves into groups to create emergent organizations that no individual may intend, comprehend, or even perceive. ... two experimental paradigms in which we attempt to build predictive bridges between the beliefs, goals, and cognitive capacities of individuals and patterns of behavior at the group level, showing how the members of a group dynamically allocate themselves to resources and how innovations diffuse through a social network. Agent-based computational models have provided useful explanatory and predictive accounts. Together, the models and experiments point to tradeoffs between exploration and exploitation—that is, compromises between individuals using their own innovations and using innovations obtained from their peers—and the emergence of group-level organizations..."

I will give this a fair shot, but the abstract is not promising at all.  A great fit to the one-factor model is, after all, precisely what you should expect if there are really an immense number of factors, but your measurement procedures are all crap and depend on random subsets of them.  (Perhaps I need to turn http://bactra.org/weblog/523.html into a proper paper after all.)

"A novel paradigm was developed to study the behavior of groups of networked people searching a problem space. The authors examined how different network structures affect the propagation of information in laboratory-created groups. Participants made numerical guesses and received scores that were also made available to their neighbors in the network. The networks were compared on speed of discovery and convergence on the optimal solution. One experiment showed that individuals within a group tend to converge on similar solutions even when there is an equally valid alternative solution. Two additional studies demonstrated that the optimal network structure depends on the problem space being explored, with networks that incorporate spatially based cliques having an advantage for problems that benefit from broad exploration, and networks with greater long-range connectivity having an advantage for problems requiring less exploration."

"As Burke puts it, the comparison makes sense; individual reason (or, for that matter, the individual social science study) is highly limited in its epistemic power in comparison to settled social practice. There is typically some reasonable basis for even highly perplexing social practices; and individual reason is likely to be highly misleading in many circumstances. Individually, we suffer from so many cognitive biases and defects that it is a wonder we get up in the morning; and even highly trained experts are often wrong, even in their own fields. ...
"But as an argument the comparison is flawed; the relevant comparison should not be that between settled practice and individual reason, but between settled practice and some alternative social practice (e.g., the social practice of science, with its various self-correction mechanisms), or between settled practice and some other collective judgment (e.g., the collective judgment of an assembly or a market)."

Obviously, I need to read this book.

"We present a model of social learning of both language and skills, while assuming—insofar as possible—strict autonomy, virtual embodiment, and situatedness. ... The aim of the article is to investigate what sociocognitive mechanisms agents should have in order to be able to transmit language from one generation to the next so that it can be used as a medium to transmit internalized rules that represent skill knowledge. We have performed experiments where this knowledge solves the familiar poisonous-food problem. ... we show that agents need to coordinate interactions so that they can establish joint attention in order to form a scaffold for language learning, which in turn forms a scaffold for the learning of rule-based skills. Based on these findings, we conclude by hypothesizing that social learning at one level forms a scaffold for the social learning at another, higher level, thus contributing to the accumulation of cultural knowledge."

We consider a decision network on an undirected graph in which each node corresponds to a decision variable, and each node and edge of the graph is associated with a reward function whose value depends only on the variables of the corresponding nodes. The goal is to construct a decision vector which maximizes the total reward. This decision problem encompasses a variety of models, including maximum-likelihood inference in graphical models (Markov Random Fields), combinatorial optimization on graphs, economic team theory and statistical physics. The network is endowed with a probabilistic structure in which costs are sampled from a distribution. Our aim is to identify sufficient conditions to guarantee average-case polynomiality of the underlying optimization problem. ... we prove that [in some case we can] find near optimal solutions with high probability in a decentralized way ... based on the network exhibiting a correlation decay (long-range independence) property."

"human-subject experiments on the problems of coloring (a social differentiation task) and consensus (a social agreement task) [on a network]. Both [are] coordination games, and despite their cognitive similarity, we find that ... network structure elicits opposing behavioral effects in the two problems, with increased long-distance connectivity making consensus easier for subjects and coloring harder. We investigate the influence that subjects have on their network neighbors and the collective outcome, and find that it varies considerably, beyond what can be explained by network position alone. ... strong correlations between influence and other features of individual subject behavior. ... much of the recent research in network science ... often emphasizes network topology out of the context of any specific problem and places primacy on network position, our findings highlight the potential importance of the details of tasks and individuals in social networks."

"Most work for most people in the developed world is about coordinating their actions with those of others – colleagues, partners, underlings, bosses, customers, distributors, suppliers, publicists, regulators .... Information collection and transfer, while often important and sometimes essential to the co-ordination of actions, is not usually itself the main game." Consequently, we have not an "Information Society" but a "Joint-Action Society, although this does not quite capture all that is intended." Resonates oddly with the recent Cuff, Permuter and Cover paper on "coordination capacity".

Not sure of the relevance to the "re:" paper. "knowledge acquisition as performed by multiple agents interacting as they infer, under [noise], respective models of a complex system. ... at each time step, each agent takes into account its current observation as well as the average of the models of its neighbors. The agents are connected by a network... of Erdős-Rényi or Barabási-Albert type. .. [if] one [agent] has a different [error rate] (higher or lower). ... [t]he influence of this special agent over the quality of the models inferred by the rest of the network can be substantial, varying linearly with the ... degree of the [special] agent ... [if] the degree of this agent is taken as a respective fitness parameter, the effect of the different [error rate] is ... superlinear.. when the agents are grouped into communities ... edges between agents (within a community) having higher probability of observation error [worsens] the estimation of the agents in the other communities."

"We develop elements of a theory of cooperation and coordination in networks. Rather than considering a communication network as a means of distributing information, or of reconstructing random processes at remote nodes, we ask what dependence can be established among the nodes given the communication constraints. Specifically, in a network with communication rates {R_{i,j}} between the nodes, we ask what is the set of all achievable joint distributions p(x1, ..., xm) of actions at the nodes of the network. Several networks are solved, including arbitrarily large cascade networks.
Distributed cooperation can be the solution to many problems such as distributed games, distributed control, and establishing mutual information bounds on the influence of one part of a physical system on another."

"After examining the careers of more than a thousand star analysts at Wall Street investment banks, and conducting more than two hundred frank interviews, Groysberg comes to a striking conclusion: star analysts who change firms suffer an immediate and lasting decline in performance. Their earlier excellence appears to have depended heavily on their former firms' general and proprietary resources, organizational cultures, networks, and colleagues. There are a few exceptions, such as stars who move with their teams and stars who switch to better firms. Female stars also perform better after changing jobs than their male counterparts do. But most stars who switch firms turn out to be meteors, quickly losing luster in their new settings." --- Favorable review in Nature.

"We use agent-based modeling to investigate the effect of conservatism and partisanship on the efficiency with which large populations solve the density classification task--a paradigmatic problem for information aggregation and consensus building. We find that conservative agents enhance the populations' ability to efficiently solve the density classification task despite large levels of noise in the system. In contrast, we find that the presence of even a small fraction of partisans holding the minority position will result in deadlock or a consensus on an incorrect answer. Our results provide a possible explanation for the emergence of conservatism and suggest that even low levels of partisanship can lead to significant social costs."

"We investigate a toy model of inductive interacting agents aiming to forecast a continuous, exogenous random variable E. Private information on E is spread heterogeneously across agents. Herding turns out to be the preferred forecasting mechanism when heterogeneity is maximal. However in such conditions aggregating information efficiently is hard even in the presence of learning, as the herding ratio rises significantly above the efficient-market expectation of 1 and remarkably close to the empirically observed values. We also study how different parameters (interaction range, learning rate, cost of information and score memory) may affect this scenario and improve efficiency in the hard phase."

with commentary by Kenneth Arrow (!)

"I develop a model of (individually rational) collective reality denial in groups, organizations
and markets. Whether participants’ tendencies toward wishful thinking reinforce or dampen
each other is shown to hinge on a simple and novel mechanism. When an agent can expect
to benefit from other’s delusions, this makes him more of a realist; when he is more likely
to suffer losses from them this pushes him toward denial, which becomes contagious." --- This is a really brilliant paper.

To put what Waldmann is saying a different way: markets might be able to aggregate many little inaccurate pieces of information, but if none of the participants actually know something, market prices will not by magic become informative.

Steve Durlauf digs in to Philip Mirowski with teeth and claws.

Review: http://bactra.org/reviews/ober-democracy-and-knowledge.html

Excellent piece on credit derivatives and the underlying institutional/cognitive problems of the markets, financial modeling, etc. Makes me extra glad I didn't agree to supervise the credit default swap thesis.

Conditions for financial markets to be useful social decision-making mechanisms != conditions under which market participants make as much money as possible.

I corresponded with Chislenko for a while in the 1990s and then we dropped out of touch - I am very surprised, and saddened, to hear what happened to him.

Short but good interview with Scott. He does a good job, I think, of explaining his ideas without using jargon.

Information cascades and path-dependence in social networks: experiments!

Apparently not.

No. Please recalibrate accordingly.

Ziman's summa of his career in studying how science actually works (after his first career doing quantum field theory in condensed matter physics)

Concludes, sensibly enough, that the only model we have for such a thing is the scientific community as a whole, not any one individual, and this has implications for the whole Singularity mythology

"The problem of actually changing the world for the better is not going to be finessed with clever "online argumentation" software. To pull off that trick you have to get your hands dirty capturing, and wielding, political power."