Last month’s creation of the Institute for New Economic Thinking should provoke some serious soul-searching among professional economists, as well as the policymakers and financiers who have relied on their models.

In particular, a much deeper debate is required about economists’ flawed theories of market “efficiency” and “rationality”, which have led economics and policy astray in the past decade, with recent disastrous results. Although financial-market practitioners now largely deride or ignore these concepts, they continue to shape the debate about fiscal stimulus, financial reform and, more broadly, the future of capitalism — which means that they remain a danger to all concerned.

Unfortunately, the assumptions that underpin these theories are largely inscrutable to anyone lacking a PhD in economics. The debate is full of terms that mean one thing to the uninitiated and quite another to economists.

Take rationality. Webster’s Dictionary defines it as reasonableness. By contrast, for economists, a rational individual is not merely reasonable; he or she is someone who behaves in accordance with a mathematical model of individual decision-making that economists have agreed to call rational.

The centrepiece of this standard of rationality, the so-called Rational Expectations Hypothesis (REH), presumes that economists can model exactly how rational individuals comprehend the future. In a bit of magical thinking, it supposes that each of the many models devised by economists provides the “true” account of how market outcomes, such as asset prices, will unfold.

The economics literature is full of different models, each assuming that it alone adequately captures how all rational market participants make decisions. Although the free-market Chicago school, neo-Keynesianism and behavioral finance, for example, are quite different in other respects, each assumes the same REH-based standard of rationality. In other words, REH-based models ignore the very raison d’être of markets: no one, as Friedrich Hayek pointed out, can have access to the totality of knowledge and information dispersed throughout the economy. Similarly, as John Maynard Keynes and Karl Popper showed, we cannot rationally predict the future course of our knowledge. Today’s models of rational decision-making simply ignore these arguments.

The most reputable investment banks and credit-rating agencies used REH finance models to “price” new derivative products. But new derivatives are not engineering breakthroughs. The use of mechanical REH models to price them assumed away the ultimate source of uncertainty in markets. Far from providing a “scientific” rationale, the valuations and ratings that they yielded were bound to have little connection to reality.

The unreasonableness of economists’ standard of rationality also helps to explain why macroeconomists of all camps and finance theorists find it so hard to account for swings and risk in asset prices. Even more pernicious, despite these difficulties, their models supposedly provide a scientific basis for judging the proper roles of the market and the State in a modern economy.

But incoherent premises lead to absurd conclusions: for example, that unfettered financial markets set asset prices nearly perfectly at their true fundamental value. If so, the State should drastically curtail its supervision of the financial system. Unfortunately, many officials worldwide came to believe this claim, known as the efficient markets hypothesis, resulting in the massive deregulation of the late 1990s and early 2000s. That made the crisis more likely, if not inevitable.

Remarkably, the same economists’ flawed standard of rationality also underpins calls for the opposite approach: active state intervention in financial markets. After all, massive state oversight is a logical implication of mathematical behavioral finance models, which, in the aftermath of the crisis, have suddenly been embraced by other economists and non-academic commentators alike.

The reason is simple: although behavioral economists have uncovered piles of evidence that market participants do not act like conventional economists would predict “rational individuals” to act, they have clung to the bogus standard of rationality underlying those predictions. They interpret their empirical findings to mean that many market participants are irrational, prone to emotion or ignore economic fundamentals for other reasons. Once these individuals dominate the “rational” participants, they push asset prices away from their “true” fundamental values.

Thus, the behavioral view suggests that swings in asset prices serve no useful social function. If the State could somehow eliminate them through massive intervention, or ban irrational players by imposing strong regulatory measures, the “rational” players could reassert their control and markets would return to their normal state of setting prices at their true values.

This is implausible, because an exact model of rational decision-making is beyond the capacity of economists — or anyone else — to formulate. Once economists recognise that they cannot explain exactly how reasonable individuals make decisions and how market outcomes unfold over time, we will no longer be stuck with two polar extremes concerning the relative roles of the market and the State.

For the most part, asset prices undergo swings because participants must cope with ever-imperfect knowledge about the fundamentals that drive prices in the first place. So long as these swings remain within reasonable bounds, the State should limit its involvement to ensuring transparency and eliminating market failures. Indeed, the failures of communism amply demonstrated financial markets’ superiority to state regulators in allocating capital.

But sometimes price swings become excessive, as recent experience painfully shows. Even accepting that policy officials must cope with ever-imperfect knowledge, they can implement measures — such as guidance ranges for asset prices and changes in capital and margin requirements that depend on whether these prices are too high or too low — to dampen excessive swings.

This combination of passive and active measures sees the State’s role as intermediate between the two extreme views that hinge on economists’ standard of rationality. It would leave financial markets to allocate capital and yet lower the economic and social costs that follow when asset-price swings become excessive, and then end, as they inevitably do, in sharp reversals.

• Roman Frydman, Professor of Economics at New York University, and Michael D. Goldberg, Professor of Economics at the University of New Hampshire, are the authors of Imperfect Knowledge Economics: Exchange Rates and Risk.

Explore Economics

• Pre-Budget Report 2008

• Budget 2009

• World Economic Forum

• The Times MPC

Times Recommends

• Lies and cover-up at MG Rover exposed by report
• High oil prices push manufacturing costs up
• Primark founder steps down as chief executive