"We give an overview of statistical models and likelihood, together with two of its variants: penalized and hierarchical likelihood. The Kullback-Leibler divergence is referred to repeatedly, for defining the misspecification risk of a model, for grounding the likelihood and the likelihood crossvalidation which can be used for choosing weights in penalized likelihood. Families of penalized likelihood and sieves estimators are shown to be equivalent. The similarity of these likelihood with a posteriori distributions in a Bayesian approach is considered."

"Despite its normative appeal and widespread use, Bayes’ rule has two well-known limitations: first, it does not predict how agents should react to an information to which they assigned probability zero; second, a sizable empirical evidence documents how agents systematically deviate from its prescriptions by overreacting to information to which they assigned a positive but small probability. By replacing Dynamic Consistency with a novel axiom, Dynamic Coherence, we characterize an alternative updating rule that is not subject to these limitations, but at the same time coincides with Bayes’ rule for “normal” events. In particular, we model an agent with a utility function over consequences, a prior over priors ρ, and a threshold. In the first period she chooses the prior that maximizes the prior over priors ρ - a’ la maximum likelihood. As new information is revealed: if the chosen prior assigns to this information a probability above the threshold, she follows Bayes’ rule and updates it. Otherwise, she goes back to her prior over priors ρ, updates it using Bayes’ rule, and then chooses the new prior that maximizes the updated ρ. We also extend our analysis to the case of ambiguity aversion."

"Ambiguous images present a challenge to the visual system: How can uncertainty about the causes of visual inputs be represented when there are multiple equally plausible causes? A Bayesian ideal observer should represent uncertainty in the form of a posterior probability distribution over causes. However, in many real-world situations, computing this distribution is intractable and requires some form of approximation. We argue that the visual system approximates the posterior over underlying causes with a set of samples and that this approximation strategy produces perceptual multistability—stochastic alternation between percepts in consciousness. Under our analysis, multistability arises from a dynamic sample-generating process that explores the posterior through stochastic diffusion, implementing a rational form of approximate Bayesian inference known as Markov chain Monte Carlo (MCMC). We examine in detail the most extensively studied form of multistability, binocular rivalry, showing how a variety of experimental phenomena—gamma-like stochastic switching, patchy percepts, fusion, and traveling waves—can be understood in terms of MCMC sampling over simple graphical models of the underlying perceptual tasks. We conjecture that the stochastic nature of spiking neurons may lend itself to implementing sample-based posterior approximations in the brain."

(Actually, if I was going to try to model this as a Bayesian inference [and why would one _do_ that?], the more natural analogy would seem to be a Berk-style oscillation among equally good, i.e., equally wrong, hypotheses.)

"These days, statisticians often deal with complex, high dimensional datasets. Research- ers in statistics and machine learning have responded by creating many new methods for analyzing high dimensional data. However, many of these new methods depend on strong assumptions. The challenge of bringing low assumption inference to high dimen- sional settings requires new ways to think about the foundations of statistics. Traditional foundational concerns, such as the Bayesian versus frequentist debate, have become less important."

Shorter Fraser: Yes. Yes it is.
Longer Fraser: "Bayes introduced the observed likelihood function to statistical inference and provided a weight function to calibrate the parameter; he also introduced a confidence distribution on the parameter space but did not provide present justifications. Of course the names likelihood and confidence did not appear until much later: Fisher for likelihood and Neyman for confidence. Lindley showed that the Bayes and the confidence results were different when the model was not location. This paper examines the occurrence of true statements from the Bayes approach and from the confidence approach, and shows that the proportion of true statements in the Bayes case depends critically on the presence of linearity in the model; and with departure from this linearity the Bayes approach can be a poor approximation and be seriously misleading. Bayesian integration of weighted likelihood thus provides a first-order linear approximation to confidence, but without linearity can give substantially incorrect results."
The responses are worth reading, especially, of course, Larry's.

"Bayesian nonparametrics is a bit like the Catholic church : there is a fair bit of dogma, mystery, and reliance on countably infinite populations from the developing world."

"Bayesian rationality is the paradigm of rational behavior in neoclassical economics. A rational agent in an economic model is one who maximizes her subjective expected utility and consistently revises her beliefs according to Bayes’s rule. The paper raises the question of how, when and why this characterization of rationality came to be endorsed by mainstream economists. Though no definitive answer is provided, it is argued that the question is far from trivial and of great historiographic importance. The story begins with Abraham Wald’s behaviorist approach to statistics and culminates with Leonard J. Savage’s elaboration of subjective expected utility theory in his 1954 classic The Foundations of Statistics. It is the latter’s acknowledged fiasco to achieve its planned goal, the reinterpretation of traditional inferential techniques along subjectivist and behaviorist lines, which raises the puzzle of how a failed project in statistics could turn into such a tremendous hit in economics. A couple of tentative answers are also offered, involving the role of the consistency requirement in neoclassical analysis and the impact of the postwar transformation of US business schools." --- The guess about business schools at the end seems plausible.

No, no, no. I like harshing on lawyers as much as the next scientist, but actually reading the news story suggests something much more reasonable. The judge on the appeals court was not, it seems, throwing out Bayes's theorem, but rather refusing to be cowed by Bayes's theorem when the base rates and the likelihoods appearing in it are wild-ass guesses. To quote the news story Mason links to: "And so he decided that Bayes' theorem shouldn't again be used unless the underlying statistics are "firm"." This is, of course, the completely correct attitude; otherwise, the Bayesian posterior is simply without any evidential value whatsoever, and the difference between an expert stating "I'm, like, really sure" and "My posterior probability is 0.99" lies entirely in the latter's spurious precision.

"This paper studies the theoretical properties of Bayesian predictions and shows that under minimal conditions we can derive finite sample bounds for the loss incurred using Bayesian predictions under the Kullback-Leibler divergence. In particular, the concept of universality of predictions is discussed and universality is established for Bayesian predictions in a variety of settings. These include predictions under almost arbitrary loss functions, model averaging, predictions in a non-stationary environment and under model misspecification."

Do follow the links to the papers.  Shorter Fraser: except in very special and simple situations, Bayesian credible sets have demonstrably horrible coverage/confidence properties; that is, the probabilities attached to them do not tell you how often they really contain the true parameter values.  In fact, it scarcely seems to make sense to describe those numbers as "probabilities".  (I find Robert's response to Fraser's article extremely unconvincing, especially where it descends into pure aesthetics, e.g. saying that Bayes gives you an elegant and unified way of doing inference.  Well, so does referring all questions to the I Ching, but does it work?)

"the Calibrated Bayesian (CB) approach to statistical inference capitalizes on the strength of Bayesian and frequentist approaches to statistical inference. In the CB approach, inferences under a particular model are Bayesian, but frequentist methods are useful for model development and model checking. In this article the CB approach is outlined. Bayesian methods for missing data are then reviewed from a CB perspective. The basic theory of the Bayesian approach, and the closely related technique of multiple imputation, is described. Then applications of the Bayesian approach to normal models are described, both for monotone and nonmonotone missing data patterns. Sequential Regression Multivariate Imputation and Penalized Spline of Propensity Models are presented as two useful approaches for relaxing distributional assumptions."  Also http://arxiv.org/abs/1108.1917

Shorter: there are no objective priors.  Also: http://arxiv.org/abs/1108.2120

"Bayesian models of human learning are becoming increasingly popular in cognitive science. We argue that their purported confirmation largely relies on a methodology that depends on premises that are inconsistent with the claim that people are Bayesian about learning and inference. Bayesian models in cognitive science derive their appeal from their normative claim that the modeled inference is in some sense rational. Standard accounts of the rationality of Bayesian inference imply predictions that an agent selects the option that maximizes the posterior expected utility. Experimental confirmation of the models, however, has been claimed because of groups of agents that “probability match” the posterior. Probability matching only constitutes support for the Bayesian claim if additional unobvious and untested (but testable) assumptions are invoked. The alternative strategy of weakening the underlying notion of rationality no longer distinguishes the Bayesian model uniquely."

"It is widely believed that one should not become more confident that _all swans are white and all lions are brave_ simply by observing white swans. Irrelevant conjunction or "tacking" of a theory onto another is often thought problematic for Bayesianism, especially given the ratio measure of confirmation considered here... Using the ratio measure, the irrelevant conjunction is confirmed to the same degree as the relevant conjunct, which... is ideal: the irrelevant conjunct is irrelevant. Because the past's really having been as it now appears to have been is an irrelevant conjunct, present evidence confirms theories about past events only insofar as irrelevant conjunctions are confirmed. Hence the ideal of not confirming irrelevant conjunctions would imply that historical claims are not confirmed. ..."

"We study the Bernstein-von Mises (BvM) phenomenon, i.e., Bayesian credible sets and frequentist confidence regions for the estimation error coincide asymptotically, for the infinite-dimensional Gaussian white noise model governed by Gaussian prior with diagonal-covariance structure. While in parametric statistics this fact is a consequence of (a particular form of) the BvM Theorem, in the nonparametric setup, however, the BvM Theorem is known to fail even in some, apparently, elementary cases. In the present paper we show that BvM-like statements hold for this model, provided that the parameter space is suitably embedded into the support of the prior. The overall conclusion is that, unlike in the parametric setup, positive results regarding frequentist probability coverage of credible sets can only be obtained if the prior assigns null mass to the parameter space."

Big scan of the journal article.

"We investigate the choice of default priors for use with likelihood for Bayesian and frequentist inference. Such a prior is a density or relative density that weights an observed likelihood function, leading to the elimination of parameters that are not of interest and then a density-type assessment for a parameter of interest. For independent responses from a continuous model, we develop a prior for the full parameter that is closely linked to the original Bayes approach and provides an extension of the right invariant measure to general contexts. We then develop a modified prior that is targeted on a component parameter of interest and by targeting avoids the marginalization paradoxes of Dawid and co-workers. This modifies Jeffreys's prior and provides extensions to the development of Welch and Peers. ... combined to explore priors for a vector parameter of interest in the presence of a vector nuisance parameter. Examples ... illustrate the computation of the priors."

Bwahahahaha!

"The notion of confidence distribution (CD), an entirely frequentist concept, is in essence a Neymanian interpretation of Fisher's Fiducial distribution. It contains information related to every kind of frequentist inference. In this article, a CD is viewed as a distribution estimator of a parameter. This leads naturally to consideration of the information contained in CD, comparison of CDs and optimal CDs, and connection of the CD concept to the (profile) likelihood function. A formal development of a multiparameter CD is also presented." Hmmm. Relevant to the phil-of-bayes paper?

I will be fascinated to see what of this is "Bayesian".

A surprisingly weak paper, along the lines of "hey! did you realize that you can use the prior distribution to penalize things other than not fitting the data?", but I should re-read. Plus: this only makes sense if everyone always had both the old and the new paradigms in the support of their priors. ("Surprising", because Salmon was very good.)

Cute, but it feels a bit artificial.

Yay, me! (6.5 years from project inception to publication. This is faster than self-organization stuff, which took 8 years.)

Oh _very_ cute.

"This paper concerns the prescriptive function of decision analysis. Consider an agent who must choose an action yielding welfare that varies with an unknown state of nature. It is often asserted that such an agent should adhere to consistency axioms which imply that behavior can be represented as maximization of expected utility. However, our agent is not concerned the consistency of his behavior across hypothetical choice sets. He only wants to make a reasonable choice from the choice set that he actually faces. Hence, I reason that prescriptions for decision making should respect actuality. That is, they should promote welfare maximization in the choice problem the agent actually faces. Any choice respecting weak and stochastic dominance is rational from the actualist perspective."

In particular see the bit about "pure Bayesian reasoning" in the rejoinder.

Multi-author tutorial volume. Lots of the major figures contributing.

What on Earth is up with those scale-free networks towards the end?

"I hope [my philosophy of statistics is] sufficiently undogmatic not to imply that all those who may think rather differently from me are necessarily stupid. If at times I do seem dogmatic, it is because it is convenient to give my own views as unequivocally as possible."

More fun with Bayes.

"Previous research has demonstrated that Bayesian reasoning performance is improved if uncertainty information is
presented as natural frequencies rather than single-event probabilities. A questionnaire study of 342 college students
replicated this effect but also found that the performance-boosting benefits of the natural frequency presentation oc-
curred primarily for participants who scored high in numeracy. This finding suggests that even comprehension and
manipulation of natural frequencies requires a certain threshold of numeracy abilities, and that the beneficial effects of
natural frequency presentation may not be as general as previously believed."

In which Andy gets baffled by what people in the wider world mean by "Bayes". (To be clear, the utter_stupidity tag does not refer to Andy.)

Presumably related to his "making decisions in large worlds" paper.

Excuse me while I pound my head into a wall.

Note the extreme weakness of the sense in which Pearl is even "half-Bayesian"; the blessed St. Jerzy could agree with it.

Truly excellent book on mathematical models of herding, information contagion, etc. under rational choice, by a friend of family.

I realize Andy is joking (April Fool's), but he's almost perfectly managed to channel how I feel on the other 364 (or 365, depending) days of the year. (Except that I do keep up with what the Bayesians are doing, thank you.)

Is someone really going to claim that parametric bootstrapping is "Bayesian"?!? Memo to self, ask Steve about this.

People evolved to be frequentists. To be read in conjunction with Gigerenzer & Hoffrage's paper.

How do you make people look like Bayesian reasoners? By presenting them with explicitly frequentist probabilities, of course. Ought-to-be-classic paper. To be read in conjunction with Cosmides & Tooby.

A non-circular explanation of why Ockham's Razor works: a preference for simple theories helps us converge to the truth _faster_, even when the truth is complex.

"we need to look beyond Bayesian decision theory for an answer to the general problem of making rational decisions under uncertainty....assuming that the decision-maker is not able to decide mathematically undecideable propositions."

"The Ptolemaic approach has recently taken another form in psychology, as 'rational' Bayesian modeling of human judgement, for example of causal relations. "