Brendan Rodgers' men, who have lost three of their opening five Premier League fixtures, were far from their best again on Tuesday as they drew 2-2 after extra time in a pulsating encounter.
And the incredible action continued into the spot-kicks, with some players required to take second penalties after 20 of the first 22 were converted. A further seven followed suit until Middlesbrough's Albert Adomah missed the target to secure the Liverpool's progress.
It appeared a comfortable night was in the offing for Liverpool when teenage debutant Jordan Rossiter struck in the 10th minute, but Adam Reach headed the Championship side level midway through the second half to force extra time.
Suso's first Liverpool goal looked to have put his side through, but Patrick Bamford equalised from the spot in injury time of extra time before the Merseyside club finally progressed to the fourth round.
Right-back Clyne fired home from 30 yards out five minutes before half-time to ensure that Ronald Koeman's impressive start to his Southampton tenure continued with a victory over an Arsenal side that included 10 changes from the starting line-up at Aston Villa on Saturday.
Alexis Sanchez had put the home side in front with a stunning free-kick, but Arsene Wenger's wait to win this competition for the first time goes on after Dusan Tadic equalised with his first goal for the club from the penalty spot and Clyne's thunderbolt put the south-coast club through.
Gylfi Sigurdsson then added a second after the interval as he reacted first to Sylvain Distin's wayward header coming back off the crossbar before substitute Marvin Emnes fired in a third with three minutes remaining.
The defender fired home in each half to score his first goals for the club and give Mark Hughes' men their first victory at the Stadium of Light after Jozy Altidore had put Sunderland ahead.
Micky Mellon's men won at Premier League side Leicester City in the second round and they march on into the last 16 after pulling off another shock victory.
Benik Afobe helped himself to another double as MK Dons beat League One rivals Bradford City 2-0.
