The pulsating game, in which Liverpool were reduced to 10 men when Jay Spearing was sent off after 72 minutes, appeared to be heading for a draw when Reina spilled a shot from Danny Murphy and American international Dempsey scored from close range.
Fulham, outplayed in the first half, came back strongly in the second but Liverpool were on the wrong end of three decisions from referee Kevin Friend which otherwise might have turned the game their way.
Fulham defender Philippe Senderos was lucky not to concede a penalty after 59 minutes when he clattered into Charlie Adam inside the box.
The officials also ruled that Luis Suarez was offside putting the ball in the Fulham net when he looked level with the last defender, and Spearing was shown a red card for a foul on Moussa Dembele when he caught the Fulham player with his trailing leg after playing the ball first.
"It's frustrating because no-one ever tells us what the level of acceptance [from referees] is," Liverpool manager Kenny Dalglish told reporters.
"Jay had it in his mind to win the ball and he did win it, but the referee didn't think so.
"I don't think we were going to get anything tonight were we? The goal that Luis scored could have gone either way and we could have got a penalty when he got pulled down. We did not play as well as we have been playing, but it was enough to win the game.
"But we never got it, we will just have to brush ourselves down and get on with it."
SENT OFF
Fulham manager Martin Jol thought Spearing should have been sent off.
"It was a bad challenge," he said before concentrating on his team's first league win since the end of October.
"It was a very welcome three points, we needed that. We had more space after the sending-off and Clint had a good game, got us a much needed win and I am very pleased. We had the rub of the green tonight, it was an even game but we took the chance."
Liverpool, unbeaten in their previous 11 matches in all competitions, came to Craven Cottage seeking to equal a club record run of seven successive away wins, which included two at neighbouring Chelsea in the Premier League and League Cup.
In contrast, Fulham were looking to improve a poor home record in the league in which they had won just one of eight matches since Liverpool won 5-2 at Craven Cottage at the end of last season.
Liverpool went closest to a goal after 28 minutes when a lofted shot from Jordan Henderson beat Fulham goalkeeper Mark Schwarzer before hitting the inside of the post and bouncing away.
Substitute Stewart Downing also hit the post for Liverpool in the second half and Dempsey struck the crossbar before scoring.
The win lifted Fulham away from the relegation zone to 13th in the table on 15 points from 14 games while Liverpool stayed seventh on 23 points, three behind Newcastle United. Manchester City lead the standings with 38 points.
