The game was abandoned after less than half an hour.
"Milan left the pitch following racist insults on the part of a minority of the Pro Patria fans directed at [Kevin-Prince] Boateng, [M'Baye] Niang, [Urby] Emanuelson and [Sulley] Muntari," the club said on their website.
"I'm disappointed and saddened but I think that it was the right choice to not return to the pitch, out of respect to our players and all other coloured players in any other league," Milan coach Massimiliano Allegri told reporters.
"We have to stop these uncivilised gestures. Italy has to become a little more civilised and a little more intelligent.
"It's unfortunate because of the families and children who had come here to enjoy the beautiful day of sport and sun.
"We hope we have given a signal that will be followed if such acts occur again in any league. We apologise but we hope that these uncivilised acts will not happen again."
Italian football federation (FIGC) president Giancarlo Abete said the incident was "unspeakable and intolerable".
"We must react with force and without silence to isolate the few criminals that transformed a friendly match into an uproar that offends all of Italian football," he said on the federation's website (www.ficg.it).
"We were annoyed from the beginning," he said. "We gave a strong signal, although we regret it because a few have spoiled it for everyone else."
AC Milan director Umberto Gandini said on Twitter: "Very proud of the Milan players who decided to walk off the pitch today for racist abuse from few idiots! No racism, no stupidity."
