Uruguay defender Lugano headed home from a Javier Pastore cross three minutes into injury-time after the Argentine playmaker's opener had been cancelled out by Abou Maiga in the second half.
"We must play better," Ancelotti told a news conference after PSG went through to the last 32, where they will travel to fourth division side Sable sur Sarthe.
"I was upset because Pastore had the opportunity to score a second goal. Same for Nene. When you have the opportunity to score, you must score."
Ancelotti, who took over from Antoine Kombouare late last month, admitted he would need time to improve PSG's game.
"We have to work hard to improve. I don't know how much time it will take," he said.
"When you're 1-0 up you need in that kind of game to be able to add a second," PSG midfielder Christophe Jallet told Eurosport . "It was not a great game from us but the most important is to qualify."
PSG survived an early scare as Maiga hit the bar after only four minutes and took control of the game although they struggled to break the deadlock.
Pastore finally found the back of the net eight minutes into the second half from close range but missed two clear chances shortly afterwards.
Maiga converted a penalty after Fabrice Peru was fouled in the box by Mamadou Sakho, only for Lugano to stab in the winner deep in stoppage-time.
Toulouse earlier became the latest Ligue 1 cup casualty when they were knocked out 1-0 by third division team GFCO Ajaccio.
Mickael Colloredo struck two minutes into the second half for the home side as Toulouse joined Caen, Sochaux, Lorient and Stade Brest on the list of top flight teams eliminated by lower league opposition.
Lisandro Lopez netted a first-half hat-trick to give Olympique Lyon a 3-1 win at neighbours Lyon-la-Duchere.
Montpellier easily went past minnows Prix les Mezieres, who play in the sixth division, with a 4-0 away win.
On Monday, Monaco, who were relegated from Ligue 1 at the end of last season, will travel to fellow Ligue 2 side Angers.
