The man beaten to the FIFA presidency by Sepp Blatter in 1998, Lennart Johansson, believes he should stand aside.
Blatter caused uproar late last year when he suggested racial incidents on the pitch should be dealt with by a handshake between those involved.
He was re-elected as FIFA president in 2011 unopposed, securing a fourth term when rival Mohamed Bin Hammam was suspended from football activity over bribery allegations.
Swede Johansson, a former UEFA president, said Blatter's autocratic style was not in FIFA's best interest.
"It cannot be that only one man should be dictating and taking all the decisions about world football," Johansson said.
"There is not much more to do than to get rid of the man in question."
"I just don't like his dictatorship. Some will respect him but the majority will say he's not good. It is not very easy to admire or respect him."
Johansson said former France captain Michel Platini, the man who succeeded him as head of UEFA in 2007, would make the ideal replacement for Blatter.
"He (Platini) is to me closer to what I was looking for," Johansson said.
"He has tried to be fair, he's open-minded, he allows discussions and he obeys decisions made by the majority."