He told BBC Radio Kent: "I've heard the rumours that I fell out with the chairman and the manager.
"That's rubbish. I left because I felt it was the right time for a change. It's as simple as that."
He added: "I don't know where these rumours came from, but I want to stress that they are not true.
"I enjoyed working with the chairman Paul Scally and former manager Ronnie Jepson and Mark Stimson. I left on the best of terms.
Patterson, who made 124 appearances for the club between 1997 and 2002, enjoys folk-hero status with Gillingham supporters. In October 1999 he famously broke his leg making a goal-line clearance to prevent arch-rivals Millwall scoring at Priestfield.
After a spell as assistant manager of Folkestone, he returned to Gillingham in November 2006 to run the club's academy before being appointed youth team manager.
He leaves a valuable legacy at Gillingham, having developed youngsters of the calibre Luke Rooney, Andy Pugh, Jack Payne, Tom Wynter and Luis Cumbers, all of whom have featured in the first team squad.
Of his new job with Ashford, where he links up with Wales international and ex-Gillingham top scorer Steve Lovell, he said: "It's a big challenge which I'm looking forward to.
"Hopefully we can develop youngsters who can come through the ranks and play for the first team."
Source: BBC Sport
