But one man recognised the potential in him. John Durkan.
Charlie Swan and JP McManus celebrate winning the Smurfit Champion Hurdle at Cheltenham in 1998. Matt Browne/SPORTSFILE.
On the day he separated himself from the rest, thoughts were of the man who should have stood beside him at the winner's enclosure.
Swan got choked back tears in his post-race interview declaring: "This is for John."
In recent years McManus said: "I would never have owned the horse but for him and whenever I think of Istabraq I always think of, and remember, John."
Istabraq's win in 1998 catapulted him into the public imagination and he would back up his win by retaining his crown in '99 and 2000.
The Foot and Mouth outbreak denied him an attempt at a record fifth festival win and a tendon injury saw him pulled up not long after the start in the 2002 renewal and he was retired.
"I suppose the the third Champion Hurdle was a standout moment," Swan said.
"So few horses had ever achieved it and winning it in a course record time as well. I think that was the moment he cemented himself as one of the greats.
"He was going for five festival wins in a row in 2001 when foot and mouth happened. The timing was just awful.
"His injury in 2002 was well documented. Something I'll never forget is the ovation he got after I had pulled him up after two hurdles. I think it showed just how the racing public thought of him."
Istabraq is enjoying his retirement at McManus' Martinstown Stud and remains in fine fettle despite his age.
"I visited him today. He was bucking and kicking. He is being treated like royalty and got a heap of carrots," Swan continued.
"JP would love to parade him more but he's 26 years of ago now and he gets too excited. It's not good for the insides and he can develop a bit of colic. He's a great age now and he's treated like a king at Martinstown."
*******************
Charlie Swan was only too happy to reminisce about the exploits of the great Istabraq and reflect on his own stellar career.
His dad Donald, who rode in a Grand National, moved to Ireland from England in the 1960s and bought Modreeny House with his wife Teresa,
"Most people think that the racing came from my dad's side but there were a lot of jockeys on my mum's side. I think Mam would definitely be claiming me," he said.
Teresa's grandfather was Tom Chaloner. He was champion jockey in 1863 and won won both the 2000 Guineas and the Derby on Macaroni.
His first taste of victory came in Ballinasloe as a 12-year-old.
"I enjoyed pony racing when I was younger but I was only in it for the enjoyment," he said.
Things changed three years later when Final Assault, a horse trained by hid Dad that he had broken himself, won at Naas with the teenager on its back. The horse raced in the silks of his grandmother Nina Swan.
After convincing his parents to allow him to cut short his education at Wilson’s Hospital in Multyfarnham, Co. Westmeath Swan spent summer at Dermot Weld's stable before moving to Kevin Prendergast's yard.
He had made a blistering start to his flat career when he suffered a broken leg in a schooling accident as he battled for the apprentice title in 1986 and his weight ballooned.
"I put on some weight after the broken leg and I went up to Kevin and told him that I wanted to give the National Hunt a go and he rang Dessie (Hughes) for me that day."
Two months after joining Hughes, Tom Morgan moved to England and suddenly Swan was top jockey.
With a riding weight of 9st 7lb, Swan was hot property in National Hunt racing. There was a vacuum in Irish racing and he filled it.
"I was lucky because Tommy Carmody was finishing up and at that stage Paul Carberry was just a kid so I was riding a lot of Noel Meade's horses as well and I rode for Mouse (Morris) too," he said.
Offers to move to top trainers in England were declined and Swan went on to enjoy 1,314 winners, including 17 at Cheltenham where he was leading jockey in both 1993 and 1994 and nine-times champion in Ireland.
He was always honing his craft and reckons that he would have been a better jockey if he was riding today.
"I would watch tapes back of myself and of top jockeys. I think I'd have been a better jockey if I was riding now to be honest," he said.
"The current crop of jockeys are the best there has ever been. I mean I don't think we'll ever see another Ruby Walsh. What an amazing horseman he is."
Swan sheltered himself from injury as best he could and often refused to ride horses he deemed to be a risk. By the time he rode Istabraq for a final time in 2002 he was only riding over hurdles.
"I was training at that stage and I spoke to JP and he was like, 'For the sake of a third of your winners, do you need to ride over fence?' I needed to stay fit and healthy because I had a business to run."
He still feels the affects of his riding career and it's no wonder after suffering a list of injuries including a broken nose, lost front teeth, a fractured skull, broken collar bones, fractured ribs, a broken leg, a broken foot, a broken wrist, a broken hand, cracked little finger, punctured lung, three breaks to left arm, three breaks to right arm, scarred lip, broken vertebrae and facial scarring over eye and on forehead
"I wake up some mornings with pains in my back and arms. I broke my arms six times," Swan admits.
Swan saddled over 500 winners in both codes between 1998 and 2015 including Grade One wins for One Cool Cookie and Offshore Account but called it quits because he felt his training venture was no longer viable.
"I had some great days as a trainer and it's not something I'll ever regret doing.
"It just got too tough to make money from it.
"I think the morning I went to tell the staff that it was finished was the toughest. I had to think of my family."
The Irish Horseracing Regulatory Board (IHRB) Integrity Statistics for 2017 show that just there are 93 holders of a national hunt training license remaining in the country.
"You need to be training a lot of winners.
"It's getting much more difficult for smaller trainers to survive. There are so many overheads. Many are not making a living from it, they just do it for the love of the game. It's what gets them up in the morning."
Swan is now part charged with identifying and purchasing talented racehorses in France for JP McManus. Perhaps he'll discover the next Istabraq.