Orlando Silva warned that not enough progress was being made in improving airports in the vast country which depends almost exclusively on air transport for long-distance travel.
"Today, I have a very strong concern about our airports," Silva told reporters during the Soccerex conference.
"When we think about an event of this size and that Brazil is almost a continent, and that you can only travel around by plane, then we realise that this is the biggest risk for the 2014 World Cup."
Brazil has almost no long-distance passenger trains while many main highways are in a dilapidated state.
Twelve cities will host matches, many of them thousands of kilometres apart.
The longest distance between venues, Porto Alegre in the south and Manaus in the Amazon region, is roughly 4,500 kilometres.
A recent report by the consulting firm McKinsey and Co said that, at the end of 2009, seven of Brazil's 20 principal airports were struggling with overcrowding in both passenger areas and plane berths, frequently leading to delays or cancelled flights.
The report said Brazil's airports had capacity for 126 million passengers per year, with existing demand of 111 million that is expected to rise to 146 million by 2014.
During the Cup alone, McKinsey said as many as 6 million additional travellers would pass through Brazilian airports.
The government expects to invest 6.5 billion reais ($3.8 billion) in airports by 2014, including about 5.5 billion reais for those in the 12 host cities.
However, Silva said progress has not been quick enough and criticised the government's airport authority Infraero.
"I think Infraero will have to change its conduct, behaviour and attitude, otherwise Brazil could suffer embarrassing situations in 2014," he said.
Brazil's airport safety conditions have improved since a 2007 crash that killed almost 200 people when a plane skidded off a runway in Sao Paulo's Congonhas airport.
But critics say investments have still not kept up with demand.
Passenger traffic jumped 28 percent in the first half of 2010 from a year earlier, according to Brazilian civil aviation authorities.
Brazil was chosen in 2007 to host the tournament which was earmarked for South America by FIFA under a short-lived rotation system which also brought this year's World Cup to Africa.
Airports are not the only worry with rampant crime and social problems also a concern especially in the biggest cities.
On Sunday, armed criminals blocked one of the main highways into Rio de Janeiro in broad daylight, robbed several cars and then set the vehicles alight.
None of the victims were injured although one told Brazilian television that the robbers fired shots at his car and he had to flee into some trees by the roadside.
