The Croatian goalscorer, 28, arrived at the Vicente Calderon in place of Costa, who departed for Chelsea.
Diego Simeone's men enjoyed stunning success in 2013-14, breaking Real madrid and Barcelona's duopoly in La Liga by claiming the title before falling agonisingly short of winning the UEFA Champions League.
Costa was a huge part of Atletico's meteoric rise, scoring 36 goals in all competitions last term - including 27 in the league.
"Costa is Costa, and I'm Mario Mandzukic," the former Bayern Munich man said.
"Everyone has their own style, and I'm going to follow the instructions of my new boss, El Cholo Simeone."
"I'm really pleased that El Cholo will be my new coach," Mandzukic said.
"I know his footballing past, he was a great player. He shows the same as a coach, he lives football 100 per cent. We are quite similar."
