Jonathan Fabbro had equalised for Cerro Porteno with 12 minutes left on the clock and Ortigoza completed the dramatic comeback in the second minute of added time as the visitors emerged triumphant at the Estadio Defensores del Chaco.
Leonardo Astrada's Cerro Porteno - who have won three matches in succession - are one point clear atop the table with three rounds remaining following Libertad's 1-1 draw at home to 3 de Febrero on Monday.
Guarani made the perfect start in Asuncion when Jorge Mendoza broke the deadlock in the 14th minute.
Cerro Porteno hit back, however, with former Bayern Munich striker Julio Dos Santos equalising 13 minutes later and Sergio Diaz finding the back of the net in the 34th minute as the visitors took a one-goal lead into the break.
Guarani quickly erased the deficit after half-time thanks to Federico Santander and Luis Cabral.
Santander restored parity in the 62nd minute, while Cabral put the home side ahead just three minutes later.
But that was as good as it got for Guarani, with Fabbro and Ortigoza both on target in the final stages.
