Manager Arsene Wenger's gamble of leaving out a batch of first-team regulars against the Championship side backfired as Arsenal fan Kazim-Richards struck a 72nd-minute winner to send Rovers into the quarter-finals.
Arsenal's Czech Republic midfielder Tomas Rosicky had come closest to breaking the deadlock with a thumping shot against the bar and Rovers had to survive furious pressure as the hosts belatedly showed some urgency, but it was too little too late.
"It's painful and disappointing to lose," Wenger, whose side fell to fourth-tier Bradford City on penalties in the League Cup quarter-finals, told reporters after another low point in what has become one of his most difficult seasons at the club.
"We lacked ruthlessness and calm in front of goal and didn't make enough of our corners and in the end we were vulnerable to one mistake. We weren't good enough to win the game today."
After eliminating Liverpool in the previous round, Oldham's luck seemed to have run out as strikes from Victor Anichebe and Phil Jagielka put Everton 2-1 ahead but Matt Smith's stoppage-time header sparked wild celebrations at Boundary Park.
Three Championship sides are through after Millwall ended Luton Town's hopes of becoming the first non-league club to reach the quarter-finals for 99 years with a 3-0 victory while Barnsley were comfortable 3-1 winners at MK Dons.
"It was not the sweetest of strikes," said Rovers manager Michael Appleton, with his third club in this season's FA Cup having started at Portsmouth before a brief stint at Blackpool.
"In fairness he tried to keep the ball down because it could have ended up in Row Z, and he got the luck he deserved."
