I've written quite a bit about the shortcomings of traditional economic models of markets. Most notably, such models -- typically characterized by rational agents the actions of whom lead (by assumption) to a market equilibrium of some sort -- generally fail to explain basic dynamical features of real markets. These include, most prominently, 1) a pronounced tendency in all markets to large price fluctuations reflected in "fat tailed" distributions of returns, and 2) vigorous and persistent volatility with delicate long-memory features, which is closely linked to the way that episodes of volatility "cluster" in the market.These phenomena show up in specific mathematical signatures in the statistics of price movements, and these signatures present some of the most obvious details that any decent model of markets ought to reproduce quite naturally. Models capable of doing this have only emerged in the past two decades, and only succeed by making a clear break with the neo-classical tradition of equilibrium. This short review from 2006 by economist Blake LeBaron gives a nice introduction to the motivation for this kind of work, which very much follows in the tradition of natural science (as opposed to much of modern economics) in pursuing explanations in models with make plausible assumptions and explore the conclusions which follow from them.This is a topic I'll be writing about more in the next few months (in connection with a longer term project I'm working on). Le Baron's review is a little old (5 years) and needs updating, but he describes some crucial points very effectively. A few highlights are worth mentioning:Le Baron first outlines some of what he calls the "major puzzles of financial markets," which are directly linked to the things I just mentioned. First is simply the existence of pronounced volatility:Volatility is the most obvious and probably the most important puzzle in finance. Why do financial prices and foreign exchange rates move around so much relative to other macro series both on a short term and long term basis? The difficulty of overall financial volatility was first demonstrated in Shiller (1981), and an update is in Shiller (2003). The issue has been that it is difficult to find financial or macro economic fundamentals that move around enough to justify the large swings observed in financial markets. As a potential policy problem, and an issue for long range investors, this might be the most important puzzle faced by financial modelers. As Le Baron notes, lots of models (ARCH, GARCH, etc) have been produced which reproduce the mathematical character of volatility but without even attempting to understanding its origin. Moreover, most models in traditional economics, by sticking to the view that individuals are more or less identical and have rational expectations, simply shy away from the very phenomenon needing to be modeled: The persistence of volatility in many financial markets has led to an entire industry of models, and is an area of intense interest both in academic and commercial areas. However, although there is a lot of empirical activity, the underlying microeconomic motives for volatility persistence are still not well understood. There are very few models which have even tackled this problem. This is probably due to the fact that in a homogeneous agent framework this is simply a very difficult problem.The second puzzle of markets that Le Baron lists is what I mentioned first above: the fat tails of market returns or  "excess kurtosis" in the statistical lingo:Financial returns at relatively high frequencies (less than one month) are not normally distributed. There is not much of a strong theoretical reason that they need to be, but the hope has often been that some form of the central limit theorem should drive returns close to normality when aggregated over time. Recently, a new field, Econophysics, has appeared which stresses that returns also have additional structure that can be described using power laws. The determination and testing of power laws remains a somewhat open area, and the set of processes that generate acceptable power law pictures is also not well understood.  Aside from the long memory of volatility and fat tailed returns, he also mentions the rich dynamics of trading volume, which is equally as interesting as those of prices. Unfortunately, when it comes to the dynamics of volume, Le Baron notes, "Most traditional financial models remain completely silent."Le Baron rightly points out that these rich dynamics in time are almost certainly linked directly to two things: 1) the fact that different market participants have different expectations at any moment (indeed, it is such differences which drive trading) and 2) that these differences undergo perpetual evolution through time. This suggests that a decent explanation of the origin of these so-called "stylised facts" (and others like them) will likely emerge from models which attempt to follow and capture something about the dynamics of beliefs and expectations in a population of diverse interacting agents: Many of the most puzzling results from finance deal with problems of behavioral heterogeneity, and the dynamics of heterogeneity. The study of market heterogeneity as a kind of complicated dynamic state variable that needs to be modeled is probably one of the defining features of agent-based models. Empirical features such as trading volume are directly related to the amount of heterogeneity in the market, and demand models that can speak to this issue. Other empirical features are probably indirectly related. Large moves, excess kurtosis, and market crashes all probably stem from some type of strategy correlation that keeps the law of large numbers from functioning well across the market. These changing patterns can only be explored in a framework that allows agent strategies to adapt and adjust over time... I think this point about the failure of the law of large numbers is, while obvious, still worth emphasizing. Something in the market ruins the simple picture of normal statistics, which would emerge from independent factors driving prices changes. Somehow the actions of different market participants must come to be correlated, thereby leading to non-normal fat tailed returns and long memory effects. In principle, of course, there may be many mechanisms contributing to this lack of independence (some participants following trends could be enough, for example).Now, I doubt anyone actually working in finance would find this observation anything but banal. Yet many economists seem determined to continue modelling markets as if this were not the case. I don't know enough economic history to know why the homogeneous rational expectations view has such a following, but it seems very weird to me indeed. The rest of Le Baron's review explores an example of the kind of model -- an agent based model for a market very much out of equilibrium -- which reproduces the above features quite naturally, at least qualitatively. The idea is simply to respect the fact that agents in a market are different and change their behaviour over time, adapting to what happens in the market. There is no presupposition that market prices must settle down to an equilibrium; the market does what it does as people interact and trade and try to profit as well as they can. The simple model explored here is one developed by Le Baron in 2002, but shares basic features with many other models developed by others. Agents can choose between a risky asset (a stock) and a risk free bond, and they use a variety of information to make their trading strategies: Agents chose over a set of portfolio strategies that map current asset market information into a recommended portfolio fraction of wealth in the risky asset. This fraction can vary from zero to one since short selling and borrowing are not allowed. Information includes lagged returns, dividend price ratios, and several trend indicators. Agents must evaluate rules using past performance, and it is in this dimension where they are assumed to be heterogeneous. Agents use differing amounts of past information to evaluate rules. In other words, they have different memory lengths when it comes to evaluating strategies. Some agents use 30 years worth of data, while others might use only 6 months. In this way this model implements to behavioral features. First, agents are clearly boundedly rational in that they do not attempt to determine the entirestate space of the economy, which would be unwieldy if they attempted this. Also, they are assumed to have “small sample bias” since they don’t all choose to use as much data as possible.The paper is an easy read so I won't get into much detail, but what emerges from the interaction of learning and adapting agents in a setting like this is immediately much more realistic and interesting than anything coming from traditional models. For example, the figure below shows the price of the risky asset as a function of time. In the model, there is a true equilibrium price which is made to fluctuate in a normal, Gaussian way (this price being linked to dividends). The actual price in the market is rarely at this equilibrium, but instead has large fluctuations around it, being some times far too high and at others too low, and often moving very rapidly from one point to another.  Now there are some features of this time series that don't look realistic. There seems to be a periodicity of sorts, for example. But this is a very simple model and the exciting thing is what it gets right -- easily giving a model of a market which never settles down to a prices, has persisting volatility, fat tails in returns, rich dynamics for trading volume and so on. Details can be found in the paper.  As Le Baron concludes the review,Agent-based models make more progress than other frameworks in explaining these features due to that fact that at their core […]

I wanted to respond to several insightful comments on my recent post on power laws in finance. And, after that, pose a question on the economics/finance history of financial time series that I hope someone out there might be able to help me with.First, comments:ivansml said... Why exactly is power-law distribution for asset returns inconsistent with EMH? It is trivial to write "standard" economic model where returns have fat tails, e.g. if we assume that stochastic process for dividends / firm profits has fat tails. That of course may not be very satisfactory explanation, but it still shows that EMH != normal distribution. In fact, Fama wrote about non-gaussian returns back in 1960's (and Mandelbrot before him), so the idea is not exactly new. The work you describe here is certainly useful and interesting, but pure patterns in data (or "stylized facts", as economists would call them) by themselves are not enough - we need some theory to make sense of them, and it would be interesting to hear more about contributions from econophysics in that area. James Picerno said... It's also worth pointing out that EMH, as I understand it, doesn't assume or dismiss that returns follow some specific distribution. Rather, EMH simply posits that prices reflect known information. For many years, analysts presumed that EMH implies a random distribution, but the empirical record says otherwise. But the random walk isn't a condition of EMH. Andrew Lo of MIT has discussed this point at length. The market may or may not be efficient, but it's not conditional on random price fluctuations. Separately, ivansmi makes a good point about models. You need a model to reject EMH. But that only brings you so far. Let's say we have a model of asset pricing that rejects EMH. Then the question is whether EMH or the model is wrong? That requires another model. In short, it's ultimately impossible to reject or accept EMH, unless of course you completely trust a given model. But that brings us back to square one. Welcome to economics. I actually agree with these statements. Let me try to clarify. In my post I said, referring to the fat tails in returns and 1/t decay of volatility correlations, that  "None of these patterns can be explained by anything in the standard economic theories of markets (the EMH etc)." The key word is of course "explained."The EMH has so much flexibility and is so loosely linked to real data that it is indeed consistent with these observations, as Ivansml (Mark) and James rightly point out. I think it is probably consistent with any conceivable time series of prices. But "being consistent with" isn't a very strong claim, especially if the consistency comes from making further subsidiary assumptions about how these fat tails might come from fluctuations in fundamental values. This seems like a "just so" story (even if the idea that fluctuations in fundamental values could have fat tails is not at all preposterous).The point I wanted to make is that nothing (that I know of) in traditional economics/finance (i.e. coming out of the EMH paradigm) gives a natural and convincing explanation of these statistical regularities. Such an explanation would start from simple well accepted facts about the behaviour of individuals, firms, etc., market structures and so on, and then demonstrate how -- because of certain logical consequences following from these facts and their interactions -- we should actually expect to find just these kinds of power laws, with the same exponents, etc., and in many different markets. Reading such an explanation, you would say "Oh, now I see where it comes from and how it works!"To illustrate some possibilities, one class of proposed explanations sees large market movements as having inherently collective origins, i.e. as reflecting large avalanches of trading behaviour coming out of the interactions of market participants. Early models in this class include the famous Santa Fe Institute Stock Market model developed in the mid 1990s. This nice historical summary by Blake LeBaron explores the motivations of this early agent-based model, the first of which was to include a focus on the interactions among market participants, and so go beyond the usual simplifying assumptions of standard theories which assume interactions can be ignored. As LeBaron notes, this work began in part...... from a desire to understand the impact of agent interactions and group learning dynamics in a financial setting. While agent-based markets have many goals, I see their first scientific use as a tool for understanding the dynamics in relatively traditional economic models. It is these models for which economists often invoke the heroic assumption of convergence to rational expectations equilibrium where agents’ beliefs and behavior have converged to a self-consistent world view. Obviously, this would be a nice place to get to, but the dynamics of this journey are rarely spelled out. Given that financial markets appear to thrive on diverse opinions and behavior, a first level test of rational expectations from a heterogeneous learning perspective was always needed.    I'm going to write posts on this kind of work soon looking in much more detail. This early model has been greatly extended and had many diverse offspring; a more recent review by LeBaron gives an updated view. In many such models one finds the natural emergence of power law distributions for returns, and also long-term correlations in volatility. These appear to be linked to various kinds of interactions between participants. Essentially, the market is an ecology of interacting trading strategies, and it has naturally rich dynamics as new strategies invade and old strategies, which had been successful, fall into disuse. The market never settles into an equilibrium, but has continuous ongoing fluctuations.Now, these various models haven't yet explained anything, but they do pose potentially explanatory mechanisms, which need to be tested in detail. Just because these mechanisms CAN produce the right numbers doesn't mean this is really how it works in markets. Indeed, some physicists and economists working together have proposed a very different kind of explanation for the power law with exponent 3 for the (cumulative) distribution of returns which links it to the known power law distribution of the wealth of investors (and hence the size of the trades they can make). This model sees large movements as arising in the large actions of very wealthy market participants. However, this is more than merely attributing the effect to unknown fat tails in fundamentals, as would be the case with EMH based explanations. It starts with empirical observations of tail behaviour in several market quantities and argues that these together imply what we see for market returns.There are more models and proposed explanations, and I hope to get into all this in some detail soon. But I hope this explains a little why I don't find the EMH based ideas very interesting. Being consistent with these statistical regularities is not as interesting as suggesting clear paths by which they arise.Of course, I might make one other point too, and maybe this is, deep down, what I find most empty about the EMH paradigm. It essentially assumes away any dynamics in the market. Fundamentals get changed by external forces and the theory supposes that this great complex mass of heterogenous humanity which is the market responds instantaneously to find the new equilibrium which incorporates all information correctly. So, it treats the non-market part of the world -- the weather, politics, business, technology and so on -- as a rich thing with potentially complicated dynamics. Then it treats the market as a really simply dynamical thing which just gets driven in slave fashion by the outside. This to me seems perversely unnatural and impossible to take seriously. But it is indeed very difficult to rule out with hard data. The idea can always be contorted to remain consistent with observations.Finally, another valuable comment:David K. Waltz said... In one of Taleeb's books, didn't he make mention that something cannot be proven true, only disproven? I think it was the whole swan thing - if you have an appropriate sample and count 100% white swans does not prove there are ONLY white swans, while a sample that has a black one proves that there are not ONLY white swans. Again, I agree completely. This is a basic point about science. We don't ever prove a theory, only disprove it. And the best science works by trying to find data to disprove a hypothesis, not by trying to prove it.I assume David is referring to my discussion of the empirical cubic power law for market returns. This is indeed a tentative stylized fact which seems to hold with appreciable accuracy in many markets, but there may well be markets in which it doesn't hold (or periods in which the exponent changes). Finding such deviations  would be very interesting as it might offer further clues as to the mechanism behind this phenomenon. NOW, for the question I wanted to pose. I've been doing some research on the history of finance, and there's something I can't quite understand. Here's the problem:1. Mandelbrot in the early 1960s showed that market returns had fat tails; he conjectured that they fit the so-called Stable Paretian (now called Stable Levy) distributions which have power law tails. These have the nice property (like the Gaussian) that the composition of the returns for longer intervals, built up from component Stable Paretian distributions, also has the same form. The market looks the same at different time scales.2. However, Mandelbrot noted in that same paper a shortcoming of his proposal. You can't think of returns as being independent and identically distributed (i.i.d.) over different time intervals because the volatility clusters -- high volatility predicts more to follow, and vice ve[…]

I've had no time to post recently for several reasons, mostly the urgent need to work on a book closely related to this blog. The deadline is getting closer. I hope to resume something like my previous posting frequency soon.But I would like to point everyone to a fascinating recent analysis of economists' opinions about the scientific method (that seems the best term for it, at least). Ole Rogeberg, a reader of this blog, alerted me to some work by himself and Hans Melberg in which they surveyed economists to see how much they looked to actual empirical tests of a theory's predictions in judging the value of a theory. The answer, it turns out, is -- not much. Internal consistency seems to be more important than empirical test.This even for a theory -- the theory of "rational addiction", which seeks to explain heroin addiction and other life destroying addictions as the consequence of fully rational choices on the part of individuals as they maximize their expected utility over their lifetimes -- which on the face of it seems highly unlikely, making the burden of empirical evidence (one would think) even higher. Some history. Gary Becker (Nobel Prize) of the University of Chicago is famous for his efforts to push the neo-classical framework into every last corner of human life. He (and many followers) have applied the trusted old recipe of utility maximization to understand (they claim) everything from crime to patterns of having children to addiction. You may see a slobbering shivering drunk or junkie in an alleyway in winter and think -- like most people -- there goes someone trapped in some very destructive behavioural feedback controlled by the interaction of addictive physical substances, emotions and so on. Not Becker. It's all quite rational, he argues.Now, Rogeberg and Melberg. Here's their abstract: This paper reports on results from a survey of views on the theory of rational addiction among academics who have contributed to this research. The topic is important because if the literature is viewed by its participants as an intellectual game, then policy makers should be aware of this so as not to derive actual policy from misleading models. A majority of the respondents believe the literature is a success story that demonstrates the power of economic reasoning. At the same time, they also believe the empirical evidence to be weak, and they disagree both on the type of evidence that would validate the theory and the policy implications. These results shed light on how many economists think about model building, evidence requirements and the policy relevance of their work.Now, in any area of science there are disgreements over what evidence really counts as important. I've certainly learned this from following 20 years of research on high temperature superconductivity, where every new paper with "knock down" evidence for some claim tends to be immediately countered by someone else claiming this evidence actually shows something quite different. The materials are complex as is the physics, and so far it just doesn't seem possible to bring clarity to the subject.But in high-Tc research, theorists are under no illusion that they understand. They readily admit that they have no good theory. The same attitude doesn't seem to have been common in economics. Rogeberg and Melberg have also described their survey work in this clearly written paper in a less technical style.A few more choice excerpts from their (full) paper below:The core of the causal insight claims from rational addiction research is that people behave in a certain way (i.e. exhibit addictive behavior) because they face and solve a specific type of choice problem. Yet rational addiction researchers show no interest in empirically examining the actual choice problem – the preferences, beliefs, and choice processes – of the people whose behavior they claim to be explaining. Becker has even suggested that the rational choice process occurs at some subconscious level that the acting subject is unaware of, making human introspection irrelevant and leaving us no known way to gather relevant data... The claim of causal insight, then, involves the claim that a choice problem people neither face nor would be able to solve prescribes an optimal consumption plan no one is aware of having. The gradual implementation of this unknown plan is then claimed to be the actual explanation for why people over time smoke more than they should according to the plans they actually thought they had. To quote Bertrand Russell out of context, this ‘is one of those views which are so absurd that only very learned men could possibly adopt them’ (Russell 1995, p. 110).On the nature of reasoning in rational addiction models (this is Nobel Prize winning stuff, by the way):[The addict]... looks strange because he sits down at (the first) period, surveys future income, production technologies, investment/addiction functions and consumption preferences over his lifetime to period T, maximizes the discounted value of his expected utility and decides to be an alcoholic. That’s the way he will get the greatest satisfaction out of life. (Winston 1980, p. 302)