Presenter Richard Keys and Gray, a former Scotland striker, were stood down temporarily after they were recorded questioning whether assistant referee Sian Massey "knew the offside rule".
Massey, 25, ran the line at Saturday's Wolverhampton Wanderers v Liverpool Premier League match and was widely praised for making the right call on Liverpool's first goal.
"Sky Sports today terminated its contract with Andy Gray," Sky Sports said on their website.
"The decision, which is effective immediately, was made in response to new evidence of unacceptable and offensive behaviour," it added.
"The new evidence, relating to an off-air incident that took place in December 2010, came to light after Andy Gray had already been subjected to disciplinary action for his comments of 22nd January 2011."
Barney Francis, managing director of Sky Sports, added: "After issuing a warning yesterday, we have no hesitation in taking this action after becoming aware of new information today," he added.
Massey was due to run the line at Tuesday's League Two game between Crewe Alexandra and Bradford City but has been withdrawn to avoid the media spotlight.
INTERVIEW:Andy Gray on breaking the law and wearing women's clothing
