"In evaluating prediction markets (and other crowd-prediction mechanisms), investigators have repeatedly observed a so-called "wisdom of crowds" effect, which roughly says that the average of participants performs much better than the average participant. The market price---an average or at least aggregate of traders' beliefs---offers a better estimate than most any individual trader's opinion. In this paper, we ask a stronger question: how does the market price compare to the best trader's belief, not just the average trader. We measure the market's worst-case log regret, a notion common in machine learning theory. To arrive at a meaningful answer, we need to assume something about how traders behave. We suppose that every trader optimizes according to the Kelly criteria, a strategy that provably maximizes the compound growth of wealth over an (infinite) sequence of market interactions. We show several consequences.…"

"A simplified model of natural convection, similar to the Lorenz (1963) system, is compared to computational fluid dynamics simulations in order to test data assimilation methods and better understand the dynamics of convection. The thermosyphon is represented by a long time flow simulation, which serves as a reference "truth". Forecasts are then made using the Lorenz-like model and synchronized to noisy and limited observations of the truth using data assimilation. The resulting analysis is observed to infer dynamics absent from the model when using short assimilation windows.

Furthermore, chaotic flow reversal occurrence and residency times in each rotational state are forecast using analysis data. Flow reversals have been successfully forecast in the related Lorenz system, as part of a perfect model experiment, but never in the presence of significant model error or unobserved variables. Finally, we provide new details concerning the fluid dynamical processes present in the thermosyphon during these flow reversals."

"To define oscillatory movements of securities market, we put in the non-local extension of Ito- equation for wavelet-images of random processes. It is proposed an algorithm of creation of evolutionary equation and a model of prediction of the most probable price movement path. It is carried out experimental validation of findings."

"We apply multiple testing procedures to the validation of estimated default probabilities in credit rating systems. The goal is to identify rating classes for which the probability of default is estimated inaccurately, while still maintaining a predefined level of committing type I errors as measured by the familywise error rate (FWER) and the false discovery rate (FDR). For FWER, we also consider procedures that take possible discreteness of the data resp. test statistics into account. The performance of these methods is illustrated in a simulation setting and for empirical default data."

"The major challenge in designing a discriminative learning algorithm for predicting structured data is to address the computational issues arising from the exponential size of the output space. Existing algorithms make different assumptions to ensure efficient, polynomial time estimation of model parameters. For several combinatorial structures, including cycles, partially ordered sets, permutations and other graph classes, these assumptions do not hold. In this thesis, we address the problem of designing learning algorithms for predicting combinatorial structures by introducing two new assumptions: (i) The first assumption is that a particular counting problem can be solved efficiently. The consequence is a generalisation of the classical ridge regression for structured prediction. (ii) The second assumption is that a particular sampling problem can be solved efficiently. …"

"Each time-series has its own linear trend, the directionality of a timeseries, and removing the linear trend is crucial to get the more intuitive matching results. Supporting the linear detrending in subsequence matching is a challenging problem due to a huge number of possible subsequences. In this paper we define this problem the linear detrending subsequence matching and propose its efficient index-based solution. To this end, we first present a notion of LD-windows (LD means linear detrending), which is obtained as follows: we eliminate the linear trend from a subsequence rather than each window itself and obtain LD-windows by dividing the subsequence into windows. Using the LD-windows we then present a lower bounding theorem for the index-based matching solution and formally prove its correctness.…"

"…The intrinsic underlying structure of the system is modeled by an epsilon-machine and its causal states. The decisional states are the emerging patterns corresponding to the utility function. In a complex systems perspective, these patterns thus form a partition of the lower-level system states that is defined according to the higher-level user's knowledge. The transitions between these decisional states correspond to events that lead to a change of decision. An algorithm is provided so as to estimate the states and their transitions from data. Application examples are given for hidden model reconstruction, cellular automata filtering, and edge detection in images."

"We give an algorithm that computes the final state of certain growth models without computing all intermediate states. Our technique is based on a "least action principle" which characterizes the odometer function of the growth process. Starting from an educated guess for the odometer, we successively correct under- and overestimates and provably arrive at the correct final state. The degree of speedup depends on the accuracy of the initial guess.
Determining the size of the boundary fluctuations in internal diffusion-limited aggregation is a long-standing open problem in statistical physics. As an application of our method, we calculate the size of fluctuations over two orders of magnitude beyond previous simulations. Our data strongly support the conjecture that the fluctuations are logarithmic in the radius."

"Motivated by a broad range of potential applications, we address the quantile prediction problem of real-valued time series. We present a sequential quantile forecasting model based on the combination of a set of elementary nearest neighbor-type predictors called "experts" and show its consistency under a minimum of conditions. Our approach builds on the methodology developed in recent years for prediction of individual sequences and exploits the quantile structure as a minimizer of the so-called pinball loss function. We perform an in-depth analysis of real-world data sets and show that this nonparametric strategy generally outperforms standard quantile prediction methods"

"The Prediction API allows you to get more from your data and makes its patterns more accessible. Specifically, the Prediction API leverages Google's machine learning infrastructure to give you the tools to better analyze your data and reveal patterns that are often difficult to manually discover. The API also enables you to use those patterns to predict new outcomes, which facilitates the development of all types of software, from textual analysis systems to recommendation systems. Because the Prediction API is a RESTful HTTP service, you can easily access it from Google App Engine, Apps Script, and other Internet-connected desktop applications."

"…Through several nu- merical experiments, we demonstrate that both matrix- and tensor-based techniques are effective for temporal link prediction despite the inherent difficulty of the problem. Additionally, we show that tensor-based techniques are particularly effective for temporal data with varying periodic patterns."

"One might almost say that the real problem isn't predicting when the earth will shake, it's organizing society so that it's not a catastrophe when that happens."

"Perhaps by coincidence, three new papers in this week’s issue of the NEP-DGE report deal with forecasting. Kolasa, Rubaszek and Skrzypczyński says that DSGE models perform remarkably well. Bache, Jore, Mitchell and Vahey claim that VAR models with structural breaks do better, but of course structural breaks cannot be predicted with a VAR. Gupta, Kabundi and Miller show that DSGE models of real estate markets are better with turning points, which are the most difficult statistic to forecast."

"We continue this protocol until the future date (1 May 2010) at which time we upload our final version of the master document. For this final version, we include the URL of a web site where the .pdf documents of all of our past forecasts can be downloaded and independently checked for consistent MD5 and SHA-2 hashes. For convenience, we will include a summary of all of our forecasts in this final document."

"Robonomics: If robots and digital systems can do everything, let them--but let human society skim value from the result. This becomes a technologically-driven version of the Basic Income Guarantee model, where citizens are given a basic above-poverty income guarantee and are free to explore education, entrepreneurship, or even a life of indolence. Or they can get one of the remaining human jobs, jobs that may pay much more than they do now in order to attract people who otherwise wouldn't want the work."

"New findings:

World industrial production continues to track closely the 1930s fall, with no clear signs of ‘green shoots’.
World stock markets have rebounded a bit since March, and world trade has stabilised, but these are still following paths far below the ones they followed in the Great Depression.
There are new charts for individual nations’ industrial output. The big-4 EU nations divide north-south; today’s German and British industrial output are closely tracking their rate of fall in the 1930s, while Italy and France are doing much worse.
The North Americans (US & Canada) continue to see their industrial output fall approximately in line with what happened in the 1929 crisis, with no clear signs of a turn around.
Japan’s industrial output in February was 25 percentage points lower than at the equivalent stage in the Great Depression. There was however a sharp rebound in March."

"When someone makes a prediction, people post it to the site along with a brief description and a URL. We monitor it and change its status to true or false when appropriate."

"Most inventors are unsuccessful, and most patents never get used. Countries that are full of inventive genius don't necessarily have booming economies. Spreading innovations is a haphazard process dependent on luck, or culture, or fickle government support... it's not a golden road to wealth and power. Innovating is an easy process compared to "un-inventing" huge installed technologies. Asbestos got yanked out of American schools, but asbestos bricks are all over the "poor world.""

"The key, then, is to have good jobs waiting for workers when they are displaced due to inevitable (and desirable) technological change or to jobs moving overseas, jobs that are every bit as good or better than the jobs they left. That is where we are falling short. "

"Foolish reliance on Li's model lead to disaster and it was made possible by CDS markets which convinced participants that they had many observations on the probability of default. They were convinced that prices revealed these probabilities because they had an insane mystical faith in the strong form efficient markets hypothesis and a schizophrenic simultaneous belief that they could beat the market."

"So if my guess is true, we either start to see less globalization, or we will gradually start seeing borders coming down over the coming decades. The twenty first century could well go down in history as the era of decline of the nation-state."

"The myth of the riskometer is alive and kicking. In spite of a large body of empirical evidence identifying the difficulties in measuring financial risk, policymakers and financial institutions alike continue to promote risk sensitivity.
The reasons may have to do with the fact that risk sensitivity is intuitively attractive, and the counter arguments complex. The crisis, however, shows us the folly of the riskometer. Let us hope that decision makers will rely on other methods."

"Pyflix is a small package written in Python that provides an easy entry point for getting up and running in the Netflix Prize competition. It combines an efficient storage scheme with an intuitive high-level API that allows contestants to focus on the real problem, the recommendation system algorithm. To get started with Pyflix, keep reading."

reproduce this using Pareto-GP?

"I want to focus on the metrics of the strategy trades. The performance statistics are below. I find them nothing less than stellar. The metrics that I found especially appealing are highlighted in green."

"The essential problem is the tragedy of the commons. Global warming and concern about CO2 emissions is a global, social problem that has extraordinary long term impacts, but when you look at it on an individual level, the marginal returns that a selfish individual can gain by ignoring the greater good far exceeds the marginal cost to that individual in the short run. In the long run, though, everyone pays more."

"The biggest problem with dystopian fiction is not its pessimism. I do think there's a serious issue about who's interests are best served by making people fear the future, but I think the biggest problem with most dystopian fiction is its laziness and derivative quality. Lazy futures act like visionary static, crackling and dirtying the signal-to-noise ratio, making it harder not only for truly insightful futures to be found, but corrupting the ability of normal people to see why those visions are worth understanding."

aka "Nothing is Miscellaneous"

"Flight to quality? Some real hedge funds are positive for the year even when the aggregate returns for the industry are negative. Performance dispersion is enormous in such a diverse universe. Several strategies have not been affected by prime brokers imploding, changes in short selling rules or the leverage lockdown. The best managed futures CTAs, global macro and options traders have been generating absolute returns throughout the equity and credit mayhem. Strategy diversification is so important since forecasting is difficult. Transitions from one market regime to another often requires a financial revolution."

"Last year, more than 16,000 companies filed 10-Ks or 10KSBs with the SEC. Assuming they come in evenly (they don't, but we'll say this for simplicity's sake), that would be more than 4,000 annual reports a quarter, or roughly 44 a day, each and every day of the year. Forget holidays, vacations, or your kids' birthdays — you've got annual reports to read!"

"Hedge funds are preparing to return between 10 per cent and 50 per cent of their assets under management to investors who want their money back at the end of yet another quarter of dire investment performance.

One prime broker said: “Many funds will have to close. There were a flood of redemption notices at the beginning of the quarter but many investors said they wouldn’t actually withdraw the money if performance improved. It hasn’t.”

One hedge fund said: “We’ve produced 15 per cent returns for 10 years. This year has been bad and our funds under management have been reduced from $2billion to just $300m. This is decimation.”"

"We have said before that this program is an inefficient, covert way to recapitalize the financial system. If I were a foreign central bank, I'd have a lot more confidence if the US imposed regulatory reform, took over dud banks, got rid of top management, and then did the good bank/bad bank split. That's a model that has worked and could be modified and improved. But for some unknown, the powers that be are refusing to employ formulas that have worked and prefer their own home-cooked brew."

Worth reading every word. Worth commenting on, I think.

"Go to parties. You can’t even start to know what you may find on the envelope of serendipity"

seeing this as the tail-end of a much longer discussion....

The tail-end of an interesting discussion.

"The mistake is believing, as we perhaps have too much, that the rating agency saber rattling means they have the will to downgrade. They don't. The very last thing they want to do is be accused of causing The End of the Financial World as We Know It."

"the distance between any two degrees of uncertainty equals the amount of evidence you would need to go from one to the other"

"So, as a rule, there seem to be no rules for when picking which class of forecasters to pick from."

Historical market data source.

"Best guess beats careful planning"