The two sides were locked at 1-1 after extra time, only for Arsenal to beat Wigan 4-2 on penalties.
Mertesacker fouled Callum McManaman just before the hour-mark, allowing Jordi Gomez to give the holders the lead from the penalty spot, but the Germany international atoned for his earlier error.
Gary Caldwell and Jack Collison had penalties saved by Arsenal's back-up goalkeeper Lukasz Fabianski, as Arsene Wenger's men kept their chances of ending a nine-year trophy drought alive.
"In the first half we were a bit timid," he told ITV.
"In the second half I think we hit the post twice and had a couple of chances so it was a good comeback.
"I think we have great character, even though a lot of people have questioned that."
Kim Kallstrom came off the bench to join Mikel Arteta, Olivier Giroud and Santi Cazorla among Arsenal's successful penalty takers.
"It was a really tough game," he told BBC Sport. "They defended really well but we pushed all the way through.
"The semi-final is all about winning. It was tough but we did it. We had to stay strong mentally and at 1-0 down it gets tough. We were focused and even though it was on penalties, we will take it.
"We have been struggling in recent weeks but you have to deal with that, it's part of the game.
"Hopefully, it's a great boost of confidence for the rest of the season."
