Ferdinand, brother of Manchester United defender Rio, told Westminster Magistrates Court that he had not realised at the time that any racist insult had been directed at him by the Chelsea player.
The high-profile case, focusing on foul language from both players, triggered the resignation in February of England's Italian manager Fabio Capello after the FA decided to strip Terry of the captain's armband for Euro 2012.
The court heard that it was only when Ferdinand got home and his then-girlfriend showed him a clip of the incident on YouTube that he believed Terry had referred to his colour among the expletives used.
He said he would have told officials at the time had he realised.
"I would have been obviously very hurt and I probably wouldn't have reacted at the time because, being a professional, you can't do that. I probably would have let the officials know what happened and dealt with it after the game," he said.
"When someone brings your colour into it, it takes it to another level and it's very hurtful."
Terry, who was also present in court, denies committing a racially aggravated public order offence and maintains he was sarcastically repeating what Ferdinand mistakenly thought he had said.
He faces a fine of 2,500 pounds if convicted at a hearing expected to last five days.
During cross-examination, Ferdinand agreed he had sworn at players in the past and been on the receiving end.
He said he was angry at Terry seeking a penalty call and there had been some barging on the pitch. He had also alluded to an alleged affair between the Chelsea player and the ex-girlfriend of former England team-mate Wayne Bridge.
Terry had been stripped of the England captaincy before the 2010 World Cup following those allegations.
Ferdinand also said he had been reluctant initially to talk to the police because the incident had happened on the field of play and was a sporting matter.
Allegations of racial abuse cast a shadow over the Premier League last season, with Liverpool's Luis Suarez banned for eight matches for abusing Manchester United's Patrice Evra during a match in October.
That case was dealt with by the football authorities rather than going to court.
