Harold is a fixture. As much a part of the furniture as the bar itself, and the series of wooden tables scattered around it; tables between which I manoeuvre oversized plates swimming in gravy to a revolving cast of customers, a skyscraper of empty glasses tucked under my arm, in those months that stretch from summer to winter and then summer again.

I am in my late teens and saving up cash to head off to university. I can think of no place better to see out my final moments in the city of my childhood than pouring pints in the beating heart of this vivid pocket of north London, wedged between Alan Bennett’s Primrose Hill and the kebab shops of Camden High Street. The Crown and Goose is an institution: the words conjure up memories of hazy Sundays meeting friends and making new ones; endless nights lost to the music, chewing off Graham Coxon’s ear as he sits nursing a pint with Suggs and that bloke who sells plastic bongs on the market. Blessedly devoid of the self-conscious goths and the swarms of tourists, the Crown and Goose is a sanctuary. A rare place that my dad and I – he who can be found at a handful of Camden’s drinking holes, left hand resting on his glass, right holding the pen above his crossword, a curl of Marlboro smoke rising – can both agree is cool.

And then there is Harold, always Harold. From almost any point in the pub, you can spot him looking out from his table in the corner, to the left of the main door. A miniature, less diplomatic Churchill in a trilby hat and jacket, muttering to himself at the newbies who float in and out, cultural voyeurs passing through a world that isn’t theirs. Even if you can’t see him, you know he is there – and woe betide anyone who tries to take his spot in the seconds between the doors swinging open at 11am and him stalking in, paper under his arm. “That’s my table,” he’ll bark if it comes to it, though it rarely will; the flicker in his eye will usually do the trick. It is a fact, as inflexible as the price of a pint of Guinness: that seat is Harold’s. It is hard to imagine him anywhere else, not that I bother trying until one night when, after hours of working his way through the barrels that leak and hum beneath the gentle throb of the bar, he stands up to leave and keels over. “I’ll take him home,” I offer, feeling as I do a deep-seated sense of duty towards men like Harold. Besides, I know enough from watching him make his way back and forth to the estate opposite – with a varying degree of ease, depending on the time of day or night – that he only lives in the flats on Delancey Street.

It is a squat block, a few storeys high, both inoffensive and unremarkable. In all the time I’ve known him, all the occasions I’ve refilled his glass – stopping to comment on the weather or the incremental changes on the street he’s lived on for all his life, which threaten to subsume the city that has always been his home – I can’t say I have ever spent a moment considering what happens once his hunched form disappears through the main door of the block that stands in view through the window from his table. If I had, I might never have let him leave.

“Where’s the light?” I ask as we reach the top of the stairs to his flat, the two of us huddled together as the rain spits across the balcony while I pat him down for his key. “No electric,” he slurs as we stumble inside, though even in the eerie half-light I can tell the darkness might be preferable; the smell is illuminating enough. “Damp, bloody damp,” Harold says, perhaps noticing me raise a protective hand to my face. It is the set of a stage-play that has been left to rot: a single-hob cooker, the empty fridge, the brown faded carpet curled at the edges and riddled with stains; colder inside than the strip of road between here and his adopted home, which glows through the window.

But you’re an old man, you can’t live like this, there are services, I rage, righteously, leaning in to plump a pillow on the sofa before noticing there isn’t one. I don’t know how long it takes for his laughter at my naivety to shake into tears. As his story unfolds through waves of anger and despair, he regales me with anecdotes of a childhood on this very street, one of several kids in a flat with a mother whose love never waned. As his thoughts turn to the wife he lost and the children “who don’t want to know”, the picture becomes less sharp, the threads harder to tie together, and it is unclear which came first – the endless hours in the pub, or the shunning of the children who no longer visit. There is an uncomfortable familiarity in the self-pitying rage that eventually gives way to oblivion, and as I leave him, after almost an hour, aching with sadness but relieved to escape, I swear to myself that I’ll do something. I’ll find his son, I’ll make the council pay attention. I’ll make it right.

But as the night rolls into day and I replay our conversation in my head, the story of the son who doesn’t want to know becomes more murky; the council that ignores my calls continues to lend its ears to another developer and, sometime later, the day comes for me to leave for university. That same day, in another pub, my dad tells me he is leaving London for a remote village in the south of France.

I send Harold a card the day I arrive in Brighton, but I don’t know if it ever reaches him. As the years pass, and my dad passes too, there are fewer reasons to return to that once familiar corner of Camden Town, until one day I pass by on the bus, and the pub is gone. Gone, too, are the old men who once counted as home this pub, and countless others like it, flattened to make way for flats that were never meant for them. It is hard to think about where they all go. So instead, I turn to my children, and say, “You see that corner there? That once was a pub…”

And I begin to tell them the story of Harold and his Churchill-style hat – but not the part about the son who doesn’t care. And for a moment I see them consider it, the image of an old man skulking along this street, his coat pulled tight against the wind. And then it is our stop, and as we jump off the bus my daughter reaches for my hand and I say, we’ll take the Tube home.

Charlotte Philby’s novel “The Most Difficult Thing” is published in July by Borough Press