"It hurts and I am still hurting because I couldn't envisage such a reaction," the 75-year-old head of world football's governing body said in a BBC interview.
"When you have done something which was not totally correct, I can only say I am sorry for all those people affected by my declarations."
Blatter sparked outrage, most notably in England where the FA is dealing with two high-profile racism allegations, when he said on Wednesday that incidents of racist abuse on the pitch should be settled with a handshake after the game.
His comments were criticised by players such as England defender Rio Ferdinand and former captain David Beckham and even brought a stinging rebuke from British Prime Minister David Cameron, who described them as "appalling".
However, Blatter said he would not quit over the remarks which have provoked back page headlines in Britain's newspapers such as "Go Now Blatter" and "Now He Must Go".
"I cannot resign," Blatter said. "Why should I?
"When you are faced with a problem you have to face the problem. To leave would be totally unfair and not compatible with my fighting spirit, my character, my energy, my personal commitment to football and my personal commitment to bring FIFA out of a period of difficulties.
"It's a serious incident, you cannot say it's not serious but we have to go on, I have to go on. Let's just see what is the reaction. This situation can only make us stronger by uniting us," added Blatter, who said FIFA has a "zero tolerance" attitude to racism.
The latest controversy surrounding the long-standing FIFA president erupted on Wednesday when in two separate television interviews he suggested racism in football was not a problem.
Asked by CNN if there was racism in the game, Blatter said: "I would deny it. There is no racism, there is maybe one of the players towards another, he has a word or a gesture which is not the correct one.
"But also the one who is affected by that, he should say that this is a game. We are in a game, and at the end of the game, we shake hands."
TWITTER EXCHANGE
His remarks provoked disbelief from Ferdinand, whose brother Anton is at the centre of a police probe after England and Chelsea captain John Terry was accused of racist abuse towards the Queen Park Rangers player during a recent Premier League game.
Liverpool's Luis Suarez, who like Terry denies wrongdoing, has been charged over alleged racist insults towards Manchester United's Patrice Evra.
Manchester United defender Rio was involved in a Twitter exchange with Blatter on Thursday, suggesting FIFA had tried to limit the damage by posting a photo of its president posing with South African government official Tokyo Sexwale, who has worked with FIFA on anti-racism initiatives.
Sexwale, who is the Minister of Human Settlements in South Africa, said Blatter's comments had been "unfortunate".
"It is important that such conduct is rooted out from the field of play witho
