Uruguayan striker Sebastian Fernandez scored a last-minute diving header at the Rosaleda, after Hercules had come back from two goals down to put themselves on the brink of qualification on the away goals rule.
The sides had drawn 0-0 in the first leg last month.
Former Real Madrid and Villarreal boss Pellegrini replaced the sacked Jesualdo Ferreira last week, and took charge of a team at the foot of the La Liga standings having lost every home game so far this season.
Elsewhere, fourth-tier Portugalete held 2007 and 2008 finalists Getafe to a 0-0 draw but went out on the away goals rule having drawn 1-1 at home in the first leg.
"It's on days like this that the coach is valued better for saying nothing than for speaking," Getafe coach Michel told reporters, after his team were whistled off at the Coliseum.
"We've gone five games without a win. There are many things that need sorting out. It's the coach's responsibility and no one else."
Valencia crushed third-tier Logrones 4-1 to progress 7-1 on aggregate and Real Mallorca drew 2-2 at Sporting Gijon to go through 5-3 on aggregate.
The last 16, first legs are scheduled to be played on December 22.
Malaga raced into a two-goal lead against Hercules with a well-worked strike from Eliseu after 10 minutes and his Portuguese compatriot Edinho netted five minutes later after intercepting a backpass.
Hercules responded when Javier Portillo struck a fierce drive following good work from Olivier Thomert down the left in the 30th minute.
Malaga had looked the more likely to score until Hercules's Abel Aguilar netted the equaliser on a break in the 62nd, but the midfielder spoiled his night by earning a second yellow card soon after to leave his side a man short.
The pressure grew on the visitors goal and Jesus Gamez crossed for Fernandez to glance his header into the far corner setting off wild celebrations.
