Martinez's side looked finished in the third round replay at Upton Park, where Enner Valencia had opened the scoring before Aiden McGeady's red card.
But Kevin Mirallas levelled with a brilliant free-kick and Romelu Lukaku put the visitors ahead in extra time.
"It is heartbreaking because you have the hardest role when you go down to 10 men for such a long period," he said.
"I thought we were the better side. We created really, really good opportunities and probably we should have taken one or two before the one we scored.
"In terms of the performance and the character and the phases of good football that we showed, I couldn't be prouder."
"Sometimes in football you feel you are going to get what you deserve but clearly we were unfortunate in the penalty shootout today, when it was a lottery," he said.
"From the support that we had from our away end, to the desire of players coming on as subs and adapting to the flexibility of the tactics, I couldn't be prouder."
