Manager Arsene Wenger's gamble of leaving out a batch of first-team regulars against the Championship side backfired as Arsenal fan Kazim-Richards struck a 72nd-minute winner to send Rovers into the quarter-finals.
Arsenal's Czech Republic midfielder Tomas Rosicky had come closest to breaking the deadlock with a thumping shot against the bar and Rovers had to survive furious pressure as the hosts belatedly showed some urgency, but it was too little too late.
"It's painful and disappointing to lose," Wenger, whose side fell to fourth-tier Bradford City on penalties in the League Cup quarter-finals, told reporters after another low point in what has become one of his most difficult seasons at the club.
"We lacked ruthlessness and calm in front of goal and didn't make enough of our corners and in the end we were vulnerable to one mistake. We weren't good enough to win the game today."
With Arsenal 21 points off the Premier League pace in fifth, a poor result at home to Bundesliga leaders Bayern Munich in their Champions League last 16, first leg on Tuesday would virtually guarantee an eighth straight trophyless campaign.
EVERTON HELD
Unusually no top flight club is definitely yet through to the FA Cup quarter-finals in what is becoming a vintage year for less-fancied sides after Everton were held 2-2 having conceded a last-gasp goal at lowly third-tier club Oldham Athletic.
After eliminating Liverpool in the previous round, Oldham's luck seemed to have run out as strikes from Victor Anichebe and Phil Jagielka put Everton 2-1 ahead but Matt Smith's stoppage-time header sparked wild celebrations at Boundary Park.
Three Championship sides are through after Millwall ended Luton Town's hopes of becoming the first non-league club to reach the quarter-finals for 99 years with a 3-0 victory while Barnsley were comfortable 3-1 winners at MK Dons.
Wenger made seven changes to the Arsenal side that won at Sunderland in the Premier League last week, leaving the likes of Jack Wilshere, Theo Walcott and Santi Cazorla on the bench.
Ironically, it was just after that trio had been introduced to try and inject some firepower to a powder-puff display with 20 minutes left that Rovers scored from a rare foray forward.
A long punt found Martin Olsson in space and when his fierce shot was only parried by Wojciech Szczesny, London-born Kazim-Richards, on loan from Turkish club Galatasaray, scuffed the ball into the ground and watched it bounce in off the post.
"It was not the sweetest of strikes," said Rovers manager Michael Appleton, with his third club in this season's FA Cup having started at Portsmouth before a brief stint at Blackpool.
"In fairness he tried to keep the ball down because it could have ended up in Row Z, and he got the luck he deserved."
NO EXCUSE
With disgruntled Arsenal fans already heading for the exits, one shout
