With Chile trailing 1-0, Bielsa took off his out-of-touch playmaker Matias Fernandez after 32 minutes and within four minutes of the move saw his team go 2-1 up.
Chile, whose last World Cup appearance was in 1998, have 30 points from 17 games and are guaranteed a top four finish in the South American group with one match to play. The defeat meant that Colombia, eighth with 20 points, were eliminated.
The top four teams in the 10-team group qualify directly for South Africa next year and the fifth plays off against a team from CONCACAF for another place.
"Matias is a player who connects the start of the move with the attack and (Jorge) Valdivia is a player who participates fully in the attack, we needed a midfielder who was less of a midfielder and more of a forward," said Bielsa.
Colombia, who have managed a measly 12 goals, went ahead after 13 minutes when Chile midfielder Arturo Vidal turned Jackson Martinez's cross into his own net.
Just after the half hour, Argentine coach Bielsa - who led his own country at the 2002 World Cup - took off Fernandez and brought on mercurial midfielder Jorge Valdiva in his place.
Three minutes later, Waldo Ponce headed Chile level and one minute after that Humberto Suazo met Fabian Orellana's low cross with a diving header to put Chile in front.
Substitute Giovanni Moreno levelled for Colombia in the 63rd minute but there was no stopping the visitors.
Valdivia clipped the ball past David Ospina to regain the lead for Chile in the 72nd minute and repaid Bielsa's faith in him seven minutes later by setting up Orellana for the fourth.
Orellana was sent off for a second bookable offence in injury-time.
