It looked as if things could not get worse for Mr Justice Peter Smith, the judge who may go down in history as the "judge who lacked judgment". He had been castigated recently by the Court of Appeal for failing to stand down in a case involving trustees where he had shown "undoubted animosity" towards one of the parties, and issued a defiant statement, digging in his heels, which at the very least made him look foolish.

Then this week the Lord Chief Justice announced that he was initiating disciplinary proceedings against the judge by referring the case to the Office for Judicial Complaints. That office now has the invidious task of recommending what action to take: at best the judge could face a reprimand; at worst, the Lord Chief Justice might have to ask the Lord Chancellor to move an address before both Houses of Parliament for his removal - the only way that a High Court judge can be dismissed from office.

Judges at their annual dinner hosted by the Lord Mayor of the City of London on Tuesday were astonished that the judge had allowed things to go so far. Several take the view that he should have resigned - and should do so now rather than see himself publicly humiliated. But Mr Justice Smith is not the run-of-the-mill High Court judge. "He is a man of extremes - and will see things as black and white," a colleague said. "He will also see this as a ganging up of the Establishment against him - and become even more resistant." He has "a kind of siege mentality", said another.

Sir Peter has form. The judge best known for The Da Vinci Code trial and his judgment, in which he inserted his own code, had already been voted the least popular Chancery judge in a straw poll of lawyers in 2003, a year after his appointment to the High Court bench. This was chiefly because of his blunt manner and the rough-and-ready way that he handled advocates. In the esoteric realms of the Chancery Division he was seen as something of a rough diamond. Plucked from the Northern Circuit where he taught law at Manchester University before entering legal practice he was appointed partly because it was decided, as one judge put it, that "we should have a Chancery judge from the North". Sir Peter is not from public school although he went to Selwyn College, Cambridge, and may have arrived at the Royal Courts of Justice aware of being an outsider. "He felt that he was different and has a bit of a chip on his shoulder as a result," one judge said.

Judging can be a lonely job and Sir Peter felt somewhat isolated. "He is away from family and friends - and the problem with this work is that you can find that you don't have the opportunity to talk to people, as you did at the Bar." A lawyer who knows him said: "I actually feel quite sorry for him. He was a good barrister in Manchester and in demand. But he has felt a bit of a fish out of water in London."

Meanwhile, some eyebrows have been raised over his judgments. The Court of Appeal was critical over his Da Vinci Code reasoning - and eyebrows were certainly raised over his inserting of a code in the judgment. "Had it been otherwise flawless, he might have got away with it," one lawyer said. "But it wasn't."

The Court of Appeal judges were unequivocal as to what they thought of the latest debacle. Sir Peter, they said, had behaved in an "intemperate" way when handling the application to stand down, had got "carried away" and indulged in "extraordinary exchanges" in court. The application that he recuse himself was "entirely justified".

The problem is that the episode displays not one but a series of errors of judgment: first, in not standing aside when the papers for the case involving trustees came before him and he saw that a partner from Addleshaw Goddard was one of the parties. He had recently been involved in unsuccessful negotiations with the firm for a job and an acrimonious e-mail exchange ensued. The exchange, revealed in court, reveals Sir Peter's bitterness at not getting the job. But judicial colleagues were also concerned at the image it portrayed of a judge seeking to ingratiate himself with a law firm, citing such comments as "the considerable advantages of being associated with someone like himself".

Then there was the error of judgment in not standing down when the application involving the Addleshaw Goddard partner came before him; and thirdly, his reaction to the Court of Appeal ruling, when he said that "as no one in the Court of Appeal or anyone else in the Ministry of Justice has yet actually told me I have been removed (although I discerned this when I prepared to sit on Tuesday but nobody turned up) and no one has given me any reasons for my removal, I am unable to comment save to say that I will not be resigning."

Sir Peter would or should have known that he must stand aside: only last year, in a case involving Mr Justice Evans-Lombe, the Court of Appeal ruled that the latter should step down because he knew someone in the case he was about to try. The court said that, if there was evidence of an apparent bias, then inconvenience, costs and delay in finding a substitute judge were not acceptable reasons for the original judge to preside. The judge should step down to avoid any possible perception of bias, the appeal judges said.

What is to be done? Judges - in this jurisdiction - rarely are reprimanded. Gary Slapper, professor of law and director of the Open University law programme, notes that "not all judges are perfect. They are no more or less universally impeccable than surgeons or government ministers". But if there are several cases of judges misbehaving in the 16th century, it is rare now. Judge Bruce Campbell is the only judge sacked in recent years - for smuggling whisky into England from Guernsey in 1983. Only one High Court judge has been removed from office on an address from both Houses of Parliament: Sir Jonah Barrington in 1830. He was found to have misappropriated money belonging to litigants.

Sir Peter had possibly fallen out of love with his work - hence his attempt to secure a job with a law firm. Now, though, such attempts are likely to have even less likelihood of success. And more bad headlines may follow: an appeal over aspects of his ruling involving the sequestrators and the property tycoon Nicholas van Hoogstraten is pending.

Meanwhile, judges are hoping that he will see sense and step aside - and finally show the judgment that has recently apparently been lacking. But whether he fights to the last or not, the episode is a cautionary tale. Sir Peter may be hung out to dry but, say more sympathetic colleagues, it should never have got to this - particularly as the judge is known to be unwell and undergoing tests. "There is no support network, no help for people when things start going wrong - and no one he would be able to turn to," said one. The system may deliver its verdict on Sir Peter but he has in turn exposed its pastoral shortcomings.

MR JUSTICE PETER SMITH: KEY FACTS

* Achieved worldwide fame after encrypting a code in his judgment in 2006 in The Da Vinci Code trial

* Fan of Admiral of the Fleet, Lord (Jackie) Fisher - hence reference to Fisher hidden in his Da Vinci Code judgment

* Ordered property tycoon Nicholas van Hoogstraten to pay £5 million to family of murdered business rival - overturned on appeal. Appeal judges described his approach as “startling”

* Hobbies: the Titanic; military history

* In 2003 voted least popular Chancery judge in a survey of lawyers published by legal magazine, Legal Business, for gruff court style

* Most likely to say: “never explain, never apologise” - first said by Lord Fisher (1841-1920)

Explore Law

• Columnists

• Law Panel

• Law Reports

• Student Law

Times Recommends

• BGC 'set aside £40m to lure Tullett brokers'
• Legal aid review to ensure value
• Fears as 21 magistrates' courts face closure