I imagine there was a light breeze as my father stepped off the boat that day. This was the Isle of Wight, after all, 1 July 1963. I picture him in a denim shirt, sleeves rolled up, leaning into a match and taking a drag of his cigarette, holding it there for a moment in that way of his, before looking up.

Did the smoke stick in his throat as he saw the words spelled out on the billboard in front of him? “Kim Philby is the Third Man”.

The world had awoken to confirmation from the prime minister, Harold Macmillan, that my grandfather, the Cambridge-educated British spy who had worked as the head of the anti-Soviet section of MI6, and principal liaison between British and US intelligence services, was all along a Communist double agent.

That was when John Philby, my father and Kim’s eldest son, also first heard the news. My dad was a 19-year-old art student at the time. It was months after Kim had boarded a freighter from Beirut and disappeared, and by then suspicions had already been aroused.

Eight years earlier, while foreign secretary, Macmillan had publicly cleared Kim of the Third Man claims after the exposure of his fellow Cambridge spies Guy Burgess and Donald Maclean. But surely, I thought, when I was old enough to think about such things, my dad must have sensed something when he was growing up. Perhaps Kim could dupe his country and his colleagues – but his family?

It was this thought that gave rise to my novel, The Most Difficult Thing, which is published this month. The issue of espionage and familial betrayal was one that had percolated in my subconscious for many years, and yet it wasn’t until I was in my early twenties that I would ask my father if he had had suspicions, while he was growing up.

It was over a bottle of red at his home in a remote pocket of south-east France, not long before he died. He replied instantly, with incredulous eyes: “Of course, I suspected he was up to something. But did I ever wonder if he was a spy?”

After my dad learnt his father had landed in Moscow, unable to set foot in the West again from behind the safety of the Iron Curtain, he took a flight to the city and tracked his father down. Over the decades, until Kim’s death in 1988, my dad went back and forth, travelling with Kim and his KGB minders and pals.

Thanks to the detailed letters Kim sent to him over the decades, which I inherited once my father died, I’ve had an intimate insight into their relationship, albeit only one side of the  conversation.

I know from these, and from the anecdotes and the family holidays in Moscow and St Petersburg before Kim died when I was five, that my father admired and loved his father hugely.

And yet, I would never have dreamt of asking my father directly how he felt about learning Kim was a double agent. He was of both a generation and breeding that favours a stiff upper lip. He spoke when he believed something was truly worth saying – and when he talked, the room quieted to hear him.

Never one to offer up unsolicited information about himself, it wasn’t until the first time he met my then boyfriend, who shared an interest in photography, that my dad mentioned his time as a war photographer for The Sunday Times, in Vietnam. After he died, I saw evidence of this, when Camera Press got in touch about the boxes of extraordinary photographic slides he took while embedded with the army.

Since his death 10 years ago, in the Languedoc village he escaped to from London in the early 2000s, I’ve been told that he was pushed off that Sunday Times job because of the associations of his name. It’s one piece in a chain of evidence that suggests bearing the moniker Philby, not least when the exposure was still so raw, would have presented some challenge.

Despite being a fiercely private person, my father’s life came under intense scrutiny. His phones were tapped. His house was watched. Reporters scrambled for interviews. Fellow (adoptive) north Londoner Alan Bennett wrote in the programme of one of his plays that my father – whom he had never met – had been late to Kim’s funeral and was drunk behind a headstone. Footage of the event, however, clearly shows him standing behind the pallbearers throughout.

When this fact was pointed out to Bennett, who wrote my father a note sticking by his false claims, my dad’s response was to simply tear the piece of paper in two and toss it in the bin. He didn’t care what others thought – and he would never be made to feel ashamed.

As open-minded as he was close-lipped (in both senses mostly very and sometimes not at all), my father believed Kim had acted upon his devotion to Communism – an ideological principal he deemed admirable against the rise of fascism. He was fiercely proud that, until the bitter end, his father stood by his principles.

Yet to John, it was neither here nor there whether he personally agreed with Kim’s actions, or the reasoning for them. What right was it of his to judge? As he liked to tell me: never explain, never apologise (though I’m afraid I don’t pay much attention to that).

Despite having no interest in setting the record straight for what he believed were published “misconceptions” about his father, knowing that I was an aspiring writer (I was a staffer at The Independent at the time of my dad’s death from lung cancer, in 2009), he encouraged me to write about Kim’s legacy as I felt fit.

Like Kim, John was nothing if not full of contradictions. He was an intellectual who disregarded formal education in order to go to art college, and become a carpenter. He was a rebel who was devoted to The Times crossword. He was an outlier who was almost universally adored. He was whip-smart, utterly charming, and he was completely unpredictable.

Monica Porter was the journalist who persuaded him to give her an exclusive interview with Daily Mail about his father, in the 90s. She wrote about the process of securing that “scoop” with my “very private” father who “didn’t trust journalists”, in the Press Gazette.

He had agreed to meet Porter, with whom he shared a mutual acquaintance, at his local wine bar. She says she struggled to get him to talk, recalling how he was “polite but totally noncommittal”. He wouldn’t be insulted with offers of money for an exclusive and told her in no uncertain terms: “I don’t give a f**k about telling my story.”

After matching him drink for drink for hours and getting thoroughly drunk as a consequence, she wrote: “Finally, in an act of desperation, I announced, ‘Look, I really need this story. If you won’t do it for any other reason, just do it for me, will you? Do it for me’.

“For a moment Philby looked at me in silence. Then he smiled slightly and shrugged. ‘OK’.”

That peculiar logic rings perfectly true.

Like his father, my dad wouldn’t be told how to think or act or feel. For those who knew him, it was his blessing and his curse. He was sharp and blunt. He could stay quiet for hours and let you drivel on, and then he could cut you dead with a single sentence. He was the most charming, the most frustrating, and the most brilliant man I’ve ever met.

As far as I’m concerned, Kim’s real betrayal – like many men before him and since – was his walking out on his children. Would my father have questioned his father leaving – in the way men do, every day, without anyone batting an eyelid – even after losing his mother at the age of 14? Probably. Would it have occurred to him to be ashamed of his father for an ideological decision he had made, and then stuck by, regardless of the personal cost? Not for a moment. And for this, I love him all the more.

‘The Most Difficult Thing‘ by Charlotte Philby (Borough Press, £12.99) is out now