Italy forward Mario Balotelli struck twice to sink the three-time European champions, who have never beaten their southern European rivals in any eight of their tournament meetings.
"The Italy hurdle proved to high for us today," said 34-year-old forward Miroslav Klose. "Maybe in the back of the head we were thinking that we have never beaten them in a tournament. It will take some time to digest this defeat."
"It is always the same thing against Italy," German football association president Wolfgang Niersbach told reporters. "At the moment it is a feeling of despair."
"Twice we were careless and twice we paid the price. After that it was difficult to turn the match against a very good team," a visibly shocked Low told reporters.
"We conceded the first goal after a cross that should never have happened and we lost our concentration. After that we let in another from a long ball where we should have closed the spaces better."
Low, who led Germany to the Euro 2008 final and a third place at the 2010 World Cup, said other than those two concentration lapses his team fought "bravely" in an effort to get back into the match.
"The air is very thin in the semi-finals and moments of carelessness can decide a game at this level and this is what happened to us," he said.
"I think we are all very disappointed but football always continues and my players will find motivation to go after new titles."
