The withdrawal places the future of the 140-year-old club back in the balance three months after going into administration. Miller said opposition to his takeover among Rangers fans had been a factor in his decision.
"We can confirm that Mr Bill Miller has decided not to complete his purchase of Rangers Football Club," joint administrator David Whitehouse of Duff & Phelps said in a statement.
"We have been informed by his advisors that there were a number of issues with which he felt uncomfortable including legacy contracts, the limitation of potential revenue streams and the expectation of required investment."
The plight of Rangers, Scottish champions a record 54 times, has sent shockwaves through British football where a number of clubs are struggling with debt.
The football season ends on Sunday, leaving Rangers facing an uncertain summer with no revenue coming in from matches.
Three other groups had expressed an interest in buying the club in recent days, Whitehouse said, holding out the prospect of a swift deal in coming days.
"There is every opportunity for these bidders to now complete the purchase of the Club prior to the end of the season," he added.
Some Rangers fans had protested about a possible takeover by Miller, who owns a tow truck business in the southern U.S. state of Tennessee.
In a statement carried by British media, Miller said that the financial situation of the club appeared to be worse than he had expected and that he was unhappy with the fans' reaction.
"After hearing the message from Rangers supporters and fans loud and clear ("Yank go home!"), I notified the administrators today that I have withdrawn my bid for Rangers and will not be moving forward," Miller said.
Miller had been named preferred bidder last week, seeing off competition from the Blue Knights consortium led by former Rangers director Paul Murray.
The club went into administration in February over nine million pounds of unpaid taxes. It faces a much larger bill relating to a tax dispute over how it paid its players over the last decade.
