Taylor converted the first spot-kick 14 minutes from time after Jason Shackell was adjudged to have brought down Kevin Nolan.
Keith Treacy was then sent off for bringing down Collison in stoppage time, the Wales international picking himself up to send Tom Heaton the wrong way.
Earlier, Kieran Trippier had rattled the West Ham crossbar with a rasping free-kick at Turf Moor.
Defeat was Burnley's first in 12 games and ended the Championship leaders' eight-match winning streak, while for a much-changed West Ham side, victory came as a welcome boost after an indifferent start to the club's Premier League campaign.
Sean Dyche made four changes to his Burnley team with three regular starters rested and David Jones missing out through suspension.
Former West Ham man Junior Stanislas was among those recalled while Sam Allardyce made nine changes from Sunday's 0-0 draw with Swansea City, only James Tomkins and Mohamed Diame retaining their starting berths.
It was Stanislas who had the game's first effort of note in the 11th minute, taking on debutant Pelly Ruddock before fizzing in a low drive from 25 yards that Adrian had to tip behind.
From the resulting corner, Stanislas fed Danny Ings inside the box, Burnley's top scorer turning and firing a shot on goal that the goalkeeper parried over the bar.
The hosts enjoyed much the better of the opening half, with Treacy in the 20th minute and Stanislas on the stroke of half-time both testing Adrian from distance, but on both occasions the Spaniard was able to move comfortably and gather the ball.
The visitors started the second half with much greater attacking intent, Tomkins seeing his goalbound header cleared in the 52nd minute, before Joe Cole's acrobatic effort was ruled out for offside a minute later.
Burnley responded in the 57th minute, Treacy thumping a volley over Adrian's bar following a clever short corner between Ings and Stanislas.
The hosts then went agonisingly close 10 minutes later, Tripper thundering a 25-yard free-kick against the crossbar with Adrian well beaten.
Second-half substitutes Stewart Downing and Collinson then combined, the latter seeing his powerful effort from close range blocked by Shackell.
Six minutes later came the game's key moment as Modibo Maiga flicked the ball on and Nolan, only just on as a substitute, fell under the challenge of Shackell and the referee pointed to the spot.
Taylor stepped up and sent his penalty to Heaton's left, the goalkeeper getting fingers to the ball on its way into the top corner.
As the hosts pressed for an equaliser, Heaton's rushed clearance fell to Collison, who looked set to fire into the unguarded net before being brought down by Treacy, who saw red for denying a goalscoring opportunity.
Collison calmly converted the penalty to seal the win and send West Ham into the quarter-finals.
