The failure of financial institutions to manage risk and the role this has played in creating today's economic crises has created awareness of the need for a new risk-management approach. This may well become the new Holy Grail of financial economics. There are two essential constituents: first, developing a more diverse mindset regarding financial risk; and second, applying more powerful techniques to capture financial reality than risk managers have used in the past.

A new method for risk management does not mean that all elements of the present system must be replaced or that all the discredited financial instruments they spawned should be discontinued. Discrimination is needed: Many existing techniques have validity, but only in certain market conditions; and innovative financial products can contribute to wealth creation, but only if they are used judiciously.

A new approach must start by understanding the reasons for the failure of risk-management systems. Why did risk models grossly understate risks? Why did managements fail to understand this? And why did regulators fail to provide effective checks and balances against the banks' risk-management mistakes?

Financial risks were understated partly because the world economy enjoyed a long period of moderate growth and low financial volatility and this “great moderation” was wrongly assumed to be a permanent benefit of prudent central banking.

But there was a more fundamental error. Risk-management models used in banks were generally based on the simplified assumption that markets fluctuated randomly following a “normal” statistical distribution. This implied very low probabilities of extreme losses, ignoring financial history. Even complex models used to price more esoteric products such as credit derivatives were based on the same weak foundations — for example, using simulations based on random numbers, which in turn assumed a normal distribution.

Many of these simplistic models grew out of Modern Portfolio Theory (MPT), which was developed half a century ago and proved highly seductive due to its simplicity and ease of computation. While MPT was useful in stable markets, it failed in extreme market conditions. More advanced techniques were developed to anticipate successions of extreme events, but these were largely ignored. Even hedge funds and investment banks that used sophisticated techniques for their trading strategies still relied on “normal” distributions for their risk management reports. This produced over-optimistic risk-return profiles and may have contributed to the excessive leverage employed.

Risk managers proved as inadequate as their models for reasons familiar from behavioural economics: emphasis was placed on precise-sounding numbers from quantitative models, while non-quantitative judgments based on experience were ignored. And although risk models produced useful numbers, managements often failed to interpret these objectively, instead applying wishful thinking. For example, extreme events such as the 1987 stock market crash, or the Long-Term Capital Management collapse in 1998, which clearly invalidated the assumptions of conventional risk models — but these were either disregarded or rationalised later as “exogenous shocks”.

Finally, there was the failure of external regulators and rating agencies to monitor the banks' risk-controls. Despite deficiencies in bank management, some of the problems could have been averted with stronger checks and balances from the regulatory bodies charged to assess risk. But rating agencies undervalued risk when assessing new products. And regulators ignored the dangers of macroeconomic contagion, as well as failing to apply any independent checks to the risk models used by banks.

One driving force behind these policy errors was competition between governments to attract financial businesses with “light-touch” regulations. Another was excessive faith among regulators in investment bankers' skills. Risk-management methods developed long ago by investment banks, for example the normally distributed Value at Risk (VAR) models, were treated by regulators as a “gold standard” — long after sophisticated financial institutions had recognised their limitations and moved on.

What reforms in risk management are needed to overcome these limitations? The key is to combine quantitative statistical methods with qualitative, or judgmental, assessments — and then to use a multi-prong strategy, involving a wide variety of methodologically independent approaches, to reach comprehensive conclusions about financial risk. When several approaches highlight similar risks, these can be flagged. Because financial uncertainty evolves in many unexpected ways, no single model will work all the time and new approaches will be required.

A good guiding principle is a “none and all” mindset. This means that no approach is ever used exclusively and all approaches are respected for what they may contribute. A simple macroeconomic example might be the ratio of debt to GDP. The growth of this ratio to historic highs in many countries was not on its own a conclusive reason to tighten regulation, but simply ignoring this warning signal violated the rule that “all” models should be considered in judging financial risks. Such qualitative judgments may create “soft” risk management boundaries, but nonetheless are important in detecting the irrational and subjective elements in market behaviour. To avoid future disasters, regulators and managers must use such “soft” qualitative approaches alongside “hard” statistical tools.

Equally urgently, the financial world must embrace new mathematical techniques that capture market reality better than standard models.

One such approach, developed at Imperial College by Nicos Christofides, uses techniques from control engineering, including neural networks and dynamic programming to produce mathematically complex but highly intuitive results. This approach is visualised through a State Transition Graph (STG) — a path of possible future price movements with probabilities assigned to each outcome. An STG can be developed for any combination of traded assets and based on long periods of historical data, including several market dislocations. STGs produce probability distributions of future price moves, which, in contrast to standard models, do not assume continuous or “normally” distributed market movements.

By mixing continuous market behaviour with discontinuous market disruptions, STG models can produce realistic probabilities of large “shocks”. They allow illiquidity risks to be considered and produce natural clusters of volatile periods. They reflect precisely the high-risk conditions that can arise in turbulent markets, but which standard models ignore. The success of STG models has been demonstrated by their performance in the credit crunch, as well as in retrospective testing on the 1987 stock market, with the statistical analysis restricted to data before each event.

The STG is only one of several innovative approaches developed in recent years by academics and sophisticated financial institutions, but largely disregarded by bank managements and regulators. Instead, traditional risk-management models have continued to be employed, even though they are known not to work. To avoid a recurrence of the present crisis, this narrow-minded attitude must change. Risk management must be reformed not by seeking a new “right” answer to replace discredited VAR models, but by creating a broader mindset — what we call the “none and all” approach.

No risk model should be relied on completely and all evidence should be taken into account. Without such an inclusive approach, even the most powerful models will fail us, at least at certain times. But with a more comprehensive understanding of risk, involving both advanced quantitative models and “soft” qualitative judgments, the chances of avoiding future financial catastrophes could be much improved.

- Shahid Chaudhri and Richard Griffiths are founding partners of Innovation4Now, a risk-management consultancy

Times Recommends

• Lies and cover-up at MG Rover exposed by report
• High oil prices push manufacturing costs up
• Primark founder steps down as chief executive