The bottom club ended a run of seven straight defeats after French-born Algerian midfielder Yebda, making his first league start, headed home a first-half Kevin Prince-Boateng cross.
"We've waited a long time for it," manager Paul Hart told Sky Sports.
Portsmouth's dismal form had seen the struggling south coast club become the first English top-flight side to lose their opening seven league games of the season for 79 years.
"It's one win and we have got to build on that and start climbing the table," Hart said.
"We played some great stuff in the first half and defended absolutely brilliantly in the second."
PAYROLL SITUATION
Hart also said the club's payroll situation would be addressed on Monday after it was announced on Thursday the first-team squad and executive board had not been paid this week.
Tottenham Hotspur rose to third in the table after twice fighting back from a goal down to draw 2-2 at Bolton Wanderers.
Niko Kranjcar's first Spurs goal and a Vedran Corluka header cancelled out a third-minute strike from Ricardo Gardner and a Kevin Davies header.
Ninth-placed Burnley made it four home wins out of four by beating Birmingham City 2-1 while Hull City put last week's 6-1 thrashing by Liverpool behind them with a 2-1 home victory over Wigan Athletic.
Birmingham keeper Joe Hart let a harmless Steven Fletcher second-half shot slip past him before a second goal by Andre Bikey meant Sebastian Larsson's late strike for the visitors was simply a consolation.
Dutch striker Jan Vennegoor of Hesselink grabbed his first Hull goal on the hour before Brazilian midfielder Geovanni added a second. Scott Sinclair netted for Wigan three minutes from time.
‘After Manchester City’s recent form, maybe they’re the underdogs against Manchester United!’ Former Red Devils defender on this weekend’s derby
‘Arteta, Alonso, Emery, me… none of us were physical players – we needed the understanding of the game. That probably helped us move into management’: Premier League boss reveals reasons for natural career progression
