So, what happened to $100-per-barrel oil? On Wednesday, the price of US light crude reached $78.77, briefly topping last summer’s record and then subsided. A few dreary bulletins about the US economy and signs that American petrol stocks were rising were enough to kill the excitement and stall the rally.

Those investors who are obsessed with the notion of the end of oil tend to forget that the price fluctuates on short-term supply considerations and signals from the underlying economy about energy demand. The oil market is not driven by guesses about how large are the Saudi Arabian reserves.

These are irrelevant on any short term or medium-term view. In the short term, the bears can point to ample oil stocks in OECD countries and economic risk on the downside. The extent of the US credit crunch and some weak jobs data together suggest a slowing American economy, which, in turn, implies less demand for oil. On the bullish side, however, Opec is asserting its discipline and, in any case, crude output from several leading Opec members is stymied by civil disturbance (Nigeria and Iraq) or lack of investment (Iran and Venezuela).

Still, the Gulf states have been investing heavily in new production and these have every interest in ensuring that the US and Chinese economies are both well supplied and continuing to depend on the dripfeed of Middle Eastern crude oil. Over the next few years, Opec will seek to keep prices strong, but not dangerously high. The oil cartel has noticed that the past few years of higher prices have not noticeably depressed demand and these prices are still below their peak of almost $90 (in inflation-adjusted terms) in 1980. It’s a reasonable guess that Opec will seek to keep its average crude price well below that discomfort level.

Does that make oil companies a good or a bad bet for an investor? Speculative exploration stocks have probably peaked for the time being. They tend to ride the oil price and the going has become more difficult for smaller companies. Drilling costs have risen to extraordinary levels; sophisticated offshore drilling rigs are hard to come by and competition for the services of top engineers and geologists is intense. At the same time, the job of the juniors has become more difficult. The easy oil has been found in the easy places. The chase has shifted to difficult oil in troublesome places. It means drilling in very deep water, technically difficult wells at high pressure and high temperatures. It also means difficult countries where smaller independent players can sometimes secure advantage if they accept political and security risks. It is the gamble taken by companies such as DNO and Addax Petroleum in the Kurdish region of Iraq.

If you are not prepared to entertain those risks, you are left with the oil majors where the nature of the game is quietly changing. These companies are gradually shifting their focus from operational output to financial outcomes.

Chasing barrels has become a mug’s game for BP, Shell and ExxonMobil. All three companies suffered flat or declining oil output in the second quarter. Exxon suffered from weak gas prices and the company’s prospects are heavily skewed to a single giant gasfield in Qatar. Shell, in particular, is stymied after a long period of weak investment and exploration performance, highlighted by its reserves scandal of three years ago. However, BP, too has come unstuck with long delays in huge oil projects, such as Atlantis and Thunder Horse in the Gulf of Mexico. In a frank appraisal of BP’s performance, Tony Hayward, the new chief executive, blamed lack of internal engineering skills and too much reliance on outsourcing. There is no easy, short-term fix to that problem.

What rescued two of the majors in the first half was their downstream business, the fuel factories that for decades were the ugly sisters of the oil fraternity. The profit margin from making petrol in America reached stratospheric levels in the first half of the year, partly because of shutdowns at BP’s US plants, including the troubled Texas City operation.

It might be tempting to dismiss big refining profits as a blip. The huge US margins of the first half will certainly decline as BP’s operations recover, but overall capacity is still constrained and will remain so for some time because of the huge political aversion to oil refiners, exacerbated by BP’s troubles. Moreover, refining is due to get more complicated. The political romance with biofuels will make the business of making petrol and diesel even more complicated and more expensive, costs that create opportunity for margin gains for astute refiners who invest cleverly.

It is no coincidence that Shell is shifting its oil sands business downstream, recognising that it is primarily a refining operation. Last year, Shell’s oil sands were the most profitable barrels in its portfolio, worth $21 each, compared with an average profit per barrel of $12. Finding the oil is not the issue anymore. It’s in Alberta, in the Gulf and in Russia. It’s how you get it to the consumer that is the opportunity and the potential for profit.

Explore Markets

• United States

• Europe

• Japan

• China

• Russia

• The Gulf

• India

• Africa

Times Recommends

• Lies and cover-up at MG Rover exposed by report
• High oil prices push manufacturing costs up
• Primark founder steps down as chief executive