The Premier League side needed an 88th minute goal from captain Thomas Vermaelen to force the extra period but failed to make their superiority tell and it was Vermaelen who missed their final spot-kick to send the minnows through.
Garry Thompson had scored with a tidy far post finish in the 16th minute to give a surprise lead to the hosts who then held off a second-half barrage before Arsenal's Belgium centre back Vermaelen levelled with a far-post header.
"We had many, many chances, their keeper had a great game," Arsenal's under pressure manager Arsene Wenger told Sky Sports. "We couldn't convert our chances. When you go out it's never good enough for Arsenal.
"I think we lost today on penalties against a team who defended well. I cannot fault the effort of our team, we gave absolutely everything for 120 minutes.
"We have to give credit to Bradford on the night. That's part of the game as well. We couldn't convert our chances for different reasons. It was a very tough game. We'll get over that it's part of our job," he added.
In the night's other quarter-final, Andreas Weimann scored two late goals as Aston Villa booked a place in the last four by coming from behind to beat Norwich City 4-1 away.
Having ruffled Arsenal's feathers, Bradford enjoyed a period of pressure before the visitors gained a foothold and began to look dangerous with Francis Coquelin firing a drive against the woodwork in the 38th and Gervinho guilty of a terrible miss.
If the luck was against the Premier League side on that occasion, it was firmly in their favour when Nahki Wells almost doubled Bradford's lead with an early shot that beat keeper Wojciech Szczesny but trickled just wide of the post.
Substitute Alex Oxlade-Chamberlain forced a good save from Duke with Arsenal's first effort on target in the 70th minute, before Vermaelen rose at the far post to head home a cross from Santi Cazorla with three minutes remaining.
Arsenal's Spain midfielder Santi Cazorla and Morocco striker Marouane Chamakh missed with the visitors' first two spot-kicks.
