The England and Manchester United defender broke his silence for the first time over the issue which has led to the Chelsea captain being stripped of the England captaincy and facing a trial in July for his alleged remarks towards the younger Ferdinand, who plays for Queens Park Rangers.
Terry has denied the charge of racially abusing Anton in a match between Chelsea and QPR at Loftus Road last October. His case will be dealt with on July 9, eight days after Euro 2012 finishes.
"Anton is my little brother," Rio told BBC's Football Focus programme. "We have grown up together and I have looked after him since we were kids. If something is going to affect him and hurt him, I am always there as a shoulder to lean on.
"In moments like this, when things are so public and you can't really say anything, it can be frustrating.
"For my family, yes, it has been tough. At the end of the day, my brother has not brought any accusations to anyone. He is not the accused.
"But he has had to sit there and take abuse from some small-minded people, which has been very disappointing.
"I hope people see what effect that has, not just on the actual person but the people around them as well, and think before they speak."
Since October, when Liverpool's Luis Suarez also racially insulted Manchester United's Patrice Evra for which he served an eight-match ban, abusive messages have appeared on Twitter and Facebook aimed at black football figures.
Ferdinand said he has thought about the numerous anti-racism initiatives that have taken place in England.
"I almost feel I have been fooled a little bit over the years," he said.
"Look back to the days when John Barnes and the other guys were playing and all the stuff they had to deal with, bananas on the pitch etc.
"I have always been someone who has championed our country for making great strides, and we have, but I thought that era was gone. It seems like it was just put to one side for a while.
"Now it seems to be a little bit more prominent. I hope it is just a group of small-minded people who are making it newsworthy at the moment and it can be stamped out."
Ferdinand said he was not interested in taking over the England armband from Terry, and was not bothered by the fact that his long-time England colleague would not be playing against United at Stamford Bridge in the Premier League on Sunday because he has a knee injury.
Asked if it was a relief Terry was not going to be involved, he said: "Not really.
"I couldn't care less if he played or not. I just think about playing for Manchester United and winning the game.
"I let the media talk about it, go on about it and create the storm. We are footballers. That is where we are best. That is where we enjoy being.
"The result and the performance of our team against theirs was definitely the most important thing for me."
