Goalkeeper Robert Green had produced a string of saves to deny the Welsh side, but Ki struck 12 minutes from time with a clinical left-foot finish before Routledge added a second to ensure Rangers' miserable form on the road continued.
Tuesday's result means Harry Redknapp's side have now lost all seven games on their travels in the Premier League following their promotion last season, scoring only two goals in the process.
Defeat was harsh on Green in particular after the former Norwich City and West Ham keeper made stunning saves to deny Gylfi Sigurdsson and Wilfried Bony in particular.
Victory for Garry Monk's side moved them into the top six, while Rangers remain in the relegation zone following their latest away loss.
Leon Britton made his first start of the season for the hosts in place of Jonjo Shelvey, while Jazz Richards came in for Angel Rangel, and Rangers centre-back Richard Dunne returned from suspension at the expense of Mauricio Isla.
Both sides had early opportunities to open the scoring, with Niko Kranjcar flashing a free-kick just wide of the Swansea goal after only four minutes.
The hosts soon got the upper hand though, and Green raced off his line to deny Routledge after Bony had sent the former QPR winger clear with an incisive pass.
Green also thwarted Kyle Bartley at the near post when the centre-back rose to meet Sigurdsson's corner with a powerful header.
Swansea remained on the front foot and Green produced a superb save down to his left on the goal-line when Sigurdsson's measured free-kick looked destined to open the scoring eight minutes before half-time.
A look at the Goal Decision System revealed just how close the Iceland midfielder came to making it 1-0.
The visitors showed very little ambition going forward, but Lukasz Fabianski had to be alert to keep out Leroy Fer's header after Kranjcar had picked the midfielder out, and Eduardo Vargas was narrowly off target with a drive from around 20 yards just before the break.
Rangers were almost caught out right at the start of the second half, but Dunne made a great last ditch tackle to ensure Bony's shot looped onto the top of the net.
Swansea fans grew increasingly frustrated as Bony's tame finish with his right foot was comfortably saved by Green after Routledge had laid the ball off invitingly for him.
The Ivory Coast striker was in the thick of the action and his well executed volley produced yet another fine stop from Green at his near post following a precise cross from Jefferson Montero.
Rangers' resistence was finally broken 12 minutes from time though, as Ki rifled a left-footed strike from a tight angle, which struck the far post before finding the back of the net.
And Routledge sealed the points with a fine right-footed finish from a Nathan Dyer cross five minutes later to heap more misery on Redknapp's men, who saw Bony come close to making it three when he struck the left-hand post late on.
