Rodrigo Tello netted the only goal for the Turkish side after 20 minutes with a shot from distance.
United, who had already qualified for the last 16, piled forward in search of an equaliser but could not find a finishing touch and were also denied late in the game by some fine saves from keeper Rustu Recber.
AC Milan were the last side to win at Old Trafford in the competition when they won 1-0 in the first knockout round in February 2005.
Besiktas, already out of contention for the knockout phase, came within a whisker of making it 2-0 in the 41st minute when a low shot by Michael Fink scraped past keeper Ben Foster's left upright but just on the wrong side of the post.
United require one more point from their remaining fixture to finish top of Group B.
They fielded a number of less familiar faces in their line-up with Gary Neville, Anderson, Ji-Sung Park, Wes Brown and Nemanja Vidic bolstering an otherwise inexperienced side.
Midfielder Gabriel Obertan and striker Daniel Welbeck played well in the first half but were given little support from the rest of the team and not even the late introduction of veteran striker Michael Owen could save United from their fate.
Besiktas coach Mustafa Denizli, speaking through a translator, praised his team's defensive display and particularly goalkeeper Rustu for his faultless performance between the posts.
"To come to England, to come to Old Trafford and be able to win, you need to have a good defence and a very good goalkeeper, Recber is just such a player," Denizli said.
United coach Alex Ferguson said the pressure of playing on the big stage may have got to his less experienced players.
BIT ANXIOUS
"It is understandable that the young players were a bit anxious. They had a few opportunities but they suffered from (conceding on) the counter-attack which can happen in European football," he said.
The Scotsman added that he was "not overly concerned" with the loss.
"You have to be fair in assessing who was playing, (losses) would be of concern if we were fielding our strongest defenders in all our games but these players (also) know the value of good defending," Ferguson said.
Besiktas were roared on by a small band of noisy supporters who largely overshadowed the majority of the 74,242 home fans with constant chanting throughout the match.
