The top-flight side may have expected to win comfortably against second-tier opposition, but they fell behind early on as goalkeeper Alessio Cragno saw a Stefano Beltrame header bounce off the post and into the net via his back.
Daniele Conti equalised from the penalty spot on the hour-mark, but Davide Luppi's fierce effort looked to have won it for the Serie B outfit.
Zdenek Zeman threw on former Italy Under-21 striker Samuele Longo in the desperate search for an equaliser and the former Inter man bundled his way through the Modena defence to grab a 93rd-minute goal.
Modena took the lead twice in the additional periods, only for Longo again and Diego Farias to peg them back, the latter with just a minute of time to be played.
The shoot-out was just as fraught, going to sudden death, before Cragno proved the hero, saving Thiago Cionek's effort - the 14th to be taken. Cagliari's reward for their thrilling win is a trip to Parma.
Sampdoria had a much easier time of it as they sailed past Brescia 2-0 at the Luigi Ferraris.
Manolo Gabbiadini opened the scoring from 12 yards after Brescia keeper Michele Arcari felled Angelo Palombo as he moved towards goal.
The second-tier club were the architects of their continued downfall as dithering defending allowed Francesco Fedato to steal in and give Gonzalo Bergessio his first Sampdoria goal via a tap-in.
