"Recent trends in planning research have led to empirical comparison becoming commonplace. The field has started to settle into a methodology for such comparisons, which for obvious practical reasons requires running a subset of planners on a subset of problems. In this paper, we characterize the methodology and examine eight implicit assumptions about the problems, planners and metrics used in many of these comparisons. The problem assumptions are: PR1) the performance of a general purpose planner should not be penalized/biased if executed on a sampling of problems and domains, PR2) minor syntactic differences in representation do not affect performance, and PR3) problems should be solvable by STRIPS capable planners unless they require ADL. The planner assumptions are: PL1) the latest version of a planner is the best one to use, PL2) default parameter settings approximate good performance, and PL3) time cut-offs do not unduly bias outcome. The metrics assumptions are: M1) performance degrades similarly for each planner when run on degraded runtime environments (e.g., machine platform) and M2) the number of plan steps distinguishes performance. We find that most of these assumptions are not supported empirically; in particular, that planners are affected differently by these assumptions. We conclude with a call to the community to devote research resources to improving the state of the practice and especially to enhancing the available benchmark problems."

"We started by scanning existing social value metrics, such as the ones described in the table “10 Ways to Measure Social Value” on page 41. We found hundreds of competing tools, of which foundations and NGOs generally use one set, governments another, and academics yet another. In addition to discovering this segmentation, our survey suggested two more reasons why so few metrics guide real decisions. First, most metrics assume that value is objective, and therefore discoverable through analysis. Yet as most modern economists now agree, value is not an objective fact. Instead, value emerges from the interaction of supply and demand, and ultimately reflects what people or organizations are willing to pay. Because so few of the tools reflect this, they are inevitably misaligned with an organization’s strategic and operational priorities."

"But if anybody starts telling you now how much they’ve beaten the S&P 500 by over the past 10 years, then before investing with them it’s definitely worth asking to see their marketing materials in the intervening years. It’s pretty important that they always used the S&P 500, and not just when it made them look good. Otherwise, Richard Ferri is right that the comparison is downright misleading, and not the kind of behavior one would expect from a fiduciary."

"The i7 even represents a sizable (35%) performance increase over the i5 model and costs only $200 more. In fact, the i7 iMac benchmarks compare favorably to 2.93Ghz Quad-Core Mac Pro which costs significantly more."

"...We find that award-winning CEOs subsequently underperform, both relative to their prior performance and relative to a matched sample of non-winning CEOs. At the same time, they extract more compensation following the awards, both in absolute amounts and relative to other top executives in their firms. They also spend more time on public and private activities outside their companies, such as assuming board seats or writing books. The incidence of earnings management increases after winning awards. The effects are strongest in firms with weak corporate governance. Our results suggest that the ex post consequences of media-induced superstar status for shareholders are negative."

"This isn't as simple as cutting out waste. The real project here is getting the medical system to define waste the same way consumers define waste: treatments that don't help people, and in fact hurt the bottom line. As it is, those treatments currently help the bottom line, and so are no more wasteful for the institution than a Best Buy salesman persuading you to buy an expensive HDMI cable you don't need."

"If we dealt with the pincer movement of lower state aid and higher enrollments by imposing admissions standards -- say, by refusing to do remediation anymore -- the economics (and prestige) of the operation would take off. Blocking developmental students would, all by itself, result in a wealthier student body. We would have much higher retention, graduation, and transfer rates. We would have much less call for special services for students with severe learning disabilities. Our financial aid spending would drop dramatically, as would our spending on tutoring. We'd run proportionally more sophomore-level classes, to the understandable delight of the faculty. As our graduation and transfer rates went up, our standing as a college of first choice would go with it. And we could both impress our politicians and insulate ourselves from them, just like the University of Michigan has. "

"What I mean by this is that over enough trades, it should not matter that the historical sequence of trades does not match exactly the real-time sequence. Regardless, it is something to keep in mind when comparing historical backtested data to real-time."

"On the outside it might appear that everybody who is not using Tool X is a moron. But speed & latency are only part of the picture. We already established that going from 5ms to 50ms might not even be noticeable by anyone using your product. The expense for speed can be multiple things:..."

Don't give a damn about the software. The benchmarks are interesting.

"While you can place some blame on high frequency traders for skewing the data, Business Insider points to week-to-week performance benchmarking as one culprit. Our sell side experience leads us to agree, even monthly performance benchmarking is ridiculous for measuring fundamental investing given the vagaries of the market in the short term. The result is that a lot of fund managers have no choice but to engage in speculative trade-chasing covered by heaps of fundamental BS to maintain their firm's fundamental, disciplined image."

"But do these goals really work? Researchers from four top business schools have collaborated to show that in many cases goals do more harm than good. Worse, they can cause real damage to organizations and individuals using them.

"We argue that the beneficial effects of goal setting have been overstated and that systematic harm caused by goal setting has been largely ignored," the researchers conclude. Bad "side effects" produced by goal-setting programs include a rise in unethical behavior, over-focus on one area while neglecting other parts of the business, distorted risk preferences, corrosion of organizational culture, and reduced intrinsic motivation."

"I want to focus on the metrics of the strategy trades. The performance statistics are below. I find them nothing less than stellar. The metrics that I found especially appealing are highlighted in green."

wow. Just... wow.

"Deregulation has not worked. Unfettered markets may produce big bonuses for CEOs, but they do not lead, as if by an invisible hand, to societal well-being. Until we achieve a better balance between markets and government, the world will continue to pay a

Useful starting point for my acceptance testing list. Noted PPV report, which is a real acceptance test based on subjective estimates; could that be fleshed out?

Among other things, testing the ability to bookmark a "related articles" link at Google Scholar.

Formalizing good habits in collaborative workflows

George muses about the Prime Directive of Retrospectives. I note that neither Retrospectives nor any of the principles of the PD are social norms in academia, even though professional development is the point there. How might that improve in future?

Some people cannot discern pedagogy from earnest geekitude. It's a framing issue: people who think "this is trivial!" are clearly making that the truth, about their dismissal of others' work... or themselves.

My wife points out that when she worked at a Big Company, the org chart was <i>protected information</I>, and not to be disclosed under any circumstances by employees.