Azzuddin who has held the position since Sept 2007 said only the FAM President Sultan Ahmad Shah has the power to sack him or ask him to step down.
"I stand by the FAM Statute whereby the appointment and termination of the secretary-general falls under the powers of the President. As such, nobody can ask me to step down or pressure me to do so. I have been carrying out my duties in accordance with the FAM Statute," he told reporters at Wisma FAM, Kelana Jaya, here, Thursday.
A local daily reported that Kelantan Football Association advisor Tan Sri Annuar Musa had allegedly asked Azzuddin to step down for failing in his duties vested upon him.
The daily said Annuar had alleged that Azzuddin failed to play his part in strengthening the relationship between the affiliates and FAM exco.
Azzuddin added that he would not be cowed by any threats from anyone who may not be happy with the action taken by the FAM Secretariat that he headed.
"This is normal and I have been going through the situation since my appointment in 2007...the pressure on the FAM Secretariat and myself is tremendous but as a former army personnel, I remain strong and face criticism hurled at me," he said.
Meanwhile, speaking of FAM's call for state associations to find their own source of funding to play in the M-League, Azzuddin said teams playing in the M-League should start finding their own financial source rather than depending on FAM.
"We will give grants to teams playing in the Super League and Premier League next year but they must find additional funding on their own. It is not FAM's responsibility to look after the finance of others," he said.
Former Perlis FA President Datuk Seri Shahidan Kassim had taken the FAM to task for not looking after the welfare of state teams that have financial problems.