Patrick Anton made the comments on Tuesday, a day after Le Graet apologised to France great Zidane for remarks about the former Real Madrid coach which drew the ire of players, politicians and the Spanish club.
Zidane was one of the favourites to take over as manager of France if Didier Deschamps left the job but the latter's contract was extended after he led the national team to the World Cup final, which they lost to Argentina last month.
When asked if Zidane, a World Cup winner with France in 1998 and a national icon, would now manage Brazil's national team instead, Le Graet told RMC: "I don't give a damn, he can go wherever he wants."
Anton told French newspaper L'Equipe: "Le Graet has made comments that show he has lost some of his lucidity. He is a man who is tired, who needs to move on.
"We need a strong and serene governance, which unfortunately is no longer the case. All season long we have to apply the rules of ethics to leaders - especially district and league presidents - and refer cases to disciplinary committees because they have crossed the line.
"As far as the president of the federation is concerned, while we obviously do not intend to refer the matter to a disciplinary committee, we can only ask him to step down in the best interests of football."
L'Equipe reported that a meeting of the FFF executive committee would take place on Wednesday.
The FFF was not immediately available to comment.
Earlier, France forward Kylian Mbappe (24) also voiced his disapproval of Le Graet's comments, saying on Twitter: "Zidane is France, we don't disrespect the legend like that."
France's Minister for Sports Amelie Oudea-Castera was one of many politicians who reacted to Le Graet's remarks, saying the president of the country's "biggest sporting federation" had crossed a line.
