In his first competitive match in charge of United, who could only finish seventh in the Premier League last term, Van Gaal saw his side struggle with his favoured 3-4-1-2 system he had used throughout pre-season.
The home side were sloppy in the first half in the opening game of the top-flight campaign on Saturday and although they improved after the break, as new captain Wayne Rooney cancelled out Ki Sung-yueng's 20-yard effort, Swansea were still able to find a winner.
Gylfi Sigurdsson was the hero for the Welsh side, powering home Wayne Routledge's cut-back with just 18 minutes remaining as United – who also saw Rooney hit the post with an excellent free-kick in the second half – started their campaign with a defeat.
The Dutchman felt his whole team were to blame for the loss and urged the club's supporters to be patient when judging his team.
Van Gaal told BT Sport: "We don't reach our level, what we have showed in the past matches, and that is disappointing because this is the most important match.
"We are disappointing [in] defence and we are disappointing ourselves. And that's why we lost today.
"Of course we need defenders, but that I have already said in former press conferences.
"You cannot say because of one game that we lost, and all the others we won, that it is the defence [that cost them].
"It is the team and how we played today. We never played as a team, I think. That is our error."
Van Gaal felt his side were much improved in the second half and refused to criticise the effort of his players.
"We scored a fantastic goal and then we have chances to make it 2-1," he added.
"But chances is not enough. At the end the result is counting. We have to evaluate why.
"They [players] have done their utmost best…the second half, they have run all the miles together. But I think we have to always keep the heads up."
Van Gaal will demand an improvement from his players when they travel to the Stadium of Light to face Sunderland next Sunday.
