WHEN Lloyd Dorfman opened his first bureau de change in Southampton Row, off London’s Holborn, he was just trying to get out of investment banking and be his own boss.

“It was a leap of faith,” he says. “I’d always dreamt of running my own business and I thought I’d rather try and fail than never try at all.”

That was in 1976. Today he sits atop Travelex, a global foreign-exchange business valued at £1 billion minimum, one of the biggest and most successful private companies in Britain. What’s even more remarkable is that he still owns two-thirds of it and keeps his head office on a couple of floors in London’s Kingsway, just round the corner from that first shop, almost as if to remind himself of the firm’s incredible journey.

“Well, it’s convenient here,” says Dorfman with a smile, “halfway between the City and the West End. And it’s easy to get to from my Hampstead home.”

He nods benignly. Tall, podgy, quietly spoken, a product of St Paul’s School in London, he is, in many ways, the most unlikely entrepreneur.

With his pinstripe suit trousers hitched high on his blue monogrammed shirt and his brown eyes blinking behind glasses on his bald, boiled-egg head, he seems, at 51, more like an amiable uncle than an avaricious empire builder.

Only when he describes his firm’s extraordinary growth as a process of “kicking down doors” do you get a sense of the determination beneath.

Dorfman has built Travelex over the years through a series of transformational deals — fighting to get into Britain’s airports, pushing into America and Australia, tying up with Abbey National in 1994, buying Thomas Cook’s financial-services arm (tripling the size of his firm) in 2001. Others have frequently failed to take his ambition seriously. It’s not a mistake anyone makes for long.

Hence the constant rumours that Dorfman must soon float his business on the London stock market, cashing in some of that £630m paper wealth. His retort is simple: what’s the hurry? He had been waiting to see what happened with the euro here, and now that seems even more unlikely — “A big thank you to the Swedes” — he’s still not sure when he’ll consider it.

The other third of his business is owned by 3i, which is content to watch its investment grow. “At some point something will happen,” he says, “but there’s no timetable.” He looks very happy keeping things just as they are.

Sitting in Travelex’s long, elegant, second-floor boardroom, studded with memorabilia — framed early traveller’s cheques, a model of the Formula One car sponsored by the firm, even two Concorde seats, propped in the corner (he just thought they’d be nice to have) — you can see why. Despite the conventional exterior, Dorfman clearly enjoys the quirks of autocratic power.

He loves the fact, for instance, that people are surprised he sponsors the Australian cricket team, rather than England.

“We were offered the world champions, we have business in Australia and, anyway, England were tied up with Vodafone,” he says.