"I could never take him totally seriously," Patten said.
Patten and his wife, Sydney, keep reminders of those years close at hand.
Of the three men, it is Bill Sr., the decent and caring man, whom Patten considers his father.
"My guess is Bill wasn't sure I was his son, but he didn't see the need to find out," Patten said.
The most intriguing thing I have heard about Patten is that he is said to be unable yet to see how Blair can be opposed.
When Patten wrote that book in the early Eighties, he was angry and fearful about the mass unemployment created by Thatcherism.
He has also stolen the voters who liked the Conservative tradition of which Patten may be the last representative.
I would prefer to think it is because Blair is creating a lasting coalition, from the Patten middle ground to, say, Robin Cook on the left.
While in Hong Kong, Patten's pay rose from pounds 152,000 to pounds 238,000 a year.
Patten, wife Lavender and their three daughters lived in Government House, a 12-bedroom mansion, with 30 servants to pamper them.
Patten had the use of a vintage Rolls Royce, two Daimlers and a helicopter.
Patten is trying to buy a six-bedroom, pounds 1million house overlooking the Thames at Barnes, South London.