The game was abandoned after less than half an hour.
"Milan left the pitch following racist insults on the part of a minority of the Pro Patria fans directed at [Kevin-Prince] Boateng, [M'Baye] Niang, [Urby] Emanuelson and [Sulley] Muntari," the club said on their website.
A video posted on YouTube showed Boateng in possession near the penalty area when jeering was heard from fans nearby.
He stopped playing, collected the ball and kicked it towards the stands. He then took his shirt off and walked off the pitch, followed by his team-mates.
"I'm disappointed and saddened but I think that it was the right choice to not return to the pitch, out of respect to our players and all other coloured players in any other league," Milan coach Massimiliano Allegri told reporters.
"We have to stop these uncivilised gestures. Italy has to become a little more civilised and a little more intelligent.
"It's unfortunate because of the families and children who had come here to enjoy the beautiful day of sport and sun.
"We hope we have given a signal that will be followed if such acts occur again in any league. We apologise but we hope that these uncivilised acts will not happen again."
Italian football federation (FIGC) president Giancarlo Abete said the incident was "unspeakable and intolerable".
"We must react with force and without silence to isolate the few criminals that transformed a friendly match into an uproar that offends all of Italian football," he said on the federation's website (www.ficg.it).
Captain Massimo Ambrosini said the jeering had started before the walkoff.
"We were annoyed from the beginning," he said. "We gave a strong signal, although we regret it because a few have spoiled it for everyone else."
Based in the town of Busto Arsizio, Pro Patria play in the fourth tier of Italian football.
AC Milan director Umberto Gandini said on Twitter: "Very proud of the Milan players who decided to walk off the pitch today for racist abuse from few idiots! No racism, no stupidity."
