The 22-year-old opened his account for the Championship club with a 14-minute second-half treble as the hosts put the relegation-threatened Premier League outfit to the sword.
Djamel Abdoun scored from the spot after 12 minutes to give Forest the lead, which was doubled midway through the second half by Paterson's first goal for the club
And the hosts' superiority was confirmed after 71 minutes as striker Paterson netted a second with a deflected effort, before claiming the match ball 11 minutes from time, with Andy Reid completing the scoring in stoppage time.
Allardyce's West Ham have just one win from their last nine matches in all competitions ahead of their League Cup semi-final first leg with Manchester City on Wednesday.
West ham struggled to mount a response, although the visitors would have levelled if not for a fine save by Karl Darlow to keep out Ravel Morrison's curling free-kick.
The hosts were left aggrieved just before the half-hour mark as they were denied another penalty following Callum Driver's rash challenge on Greg Halford.
West Ham were much improved in the early stages of the second half, Sebastian Lletget and Morrison both going close as Allardyce's side pressed for an equaliser.
Still, the hosts remained a threat and midfielder Henri Lansbury perhaps should have done better when he headed Reid's free-kick narrowly over the crossbar.
Forest refused to settle for two, though, with victory all but confirmed as Paterson's deflected effort found the net after West Ham had allowed Lansbury to waltz into the area.
