Johansson, 82, who was UEFA's president for 17 years and a former FIFA vice-president, fought and lost a bitter 1998 FIFA presidential election to Blatter, who has remained in power ever since.
Blatter, who will be 76 next month, secured a fourth term unopposed last year and has said he will retire in 2015.
However the Swede told the BBC's World Service: "It cannot be that only one man should be dictating and taking all the decisions about world football.
"There is not much more to do than to get rid of the man in question."
Johansson said that Michel Platini, who succeeded him as UEFA president in 2007, should replace Blatter.
"He is, to me, closer to what I was looking for," Johansson explained.
"He has tried to be fair, he's open-minded, he allows discussions and he obeys decisions made by the majority."
FIFA has been hit by a wave of corruption allegations over the last 18 months which have ended the footballing careers of a number of high-ranking officials including Qatari Mohammed Bin Hammam, who pulled out of the FIFA presidential election last year after being accused of bribery, allowing Blatter to be re-elected unopposed.
‘Arteta, Alonso, Emery, me… none of us were physical players – we needed the understanding of the game. That probably helped us move into management’: Premier League boss reveals reasons for natural career progression
‘England have the players to win the World Cup – it’ll be tough for Thomas Tuchel to do a bad job, with the squad he has at his disposal’ Former Three Lions winger backs new boss after gentle qualifying draw
