Paul Lambert's Villa side were on the back foot for most of the game, but showed great character after being pegged back twice to edge a five-goal thriller at St Mary's Stadium.
Gabriel Agbonlahor's first of the season set Villa on their way before Jay Rodriguez hit back just after half-time with a superb header.
Libor Kozak's third goal of the campaign restored the visitors' advantage and although Pablo Daniel Osvaldo equalised, Delph's thunderous strike won it in a breathless game.
Victory means Villa stretch their unbeaten run to five, while Southampton have now lost three successive games for the first time under Mauricio Pochettino.
Pochettino made four changes from the weekend defeat at Chelsea, two of which were enforced. Injuries to Artur Boruc and Morgan Schneiderlin meant Steven Davis and 21-year-old Paulo Gazzaniga were both recalled.
Christian Benteke, Villa's top scorer last season, dropped to the bench.
The early signs were good for Southampton as captain Adam Lallana produced a fabulous piece of skill before being thwarted by Ron Vlaar and then saw his cross deflected onto the woodwork.
Villa's goal was leading a charmed life as Rodriguez stung the palms of Brad Guzan after 13 minutes.
But Lambert's men then hit Southampton with the ultimate sucker punch a minute later. Lallana was robbed on the edge of the Villa area and Agbonlahor was released on the halfway line. He burst into the area and easily evaded Dejan Lovren before finishing with aplomb.
Nathan Baker headed well wide as the hosts went in search of a second, but Southampton remained a threat and Rodriguez was denied an equaliser through a towering back-post header by a fantastic save by Guzan, who managed to claw the ball out after 26 minutes.
Southampton, however, did get the equaliser their first-half performance deserved after 47 minutes. Nathaniel Clyne's cross was glanced into the far corner by Rodriguez.
All the pressure was from the home side and substitute Osvaldo's audacious backheel volley forced Guzan into a diving save.
Villa, though, edged ahead again after 64 minutes. Agbonlahor dispossessed Jack Cork and crossed for Kozak, whose header was too hot for Gazzaniga to handle. It was the Czech's final act before being substituted.
The lead lasted less than five minutes. A deep cross was headed across goal by Rickie Lambert into the path of Osvaldo and he nodded in from close range.
But Villa had the last laugh with 10 minutes remaining - and what a strike it was. Delph surged forward and unleashed an unstoppable strike that arrowed into the top corner.
‘After Manchester City’s recent form, maybe they’re the underdogs against Manchester United!’ Former Red Devils defender on this weekend’s derby
‘Arteta, Alonso, Emery, me… none of us were physical players – we needed the understanding of the game. That probably helped us move into management’: Premier League boss reveals reasons for natural career progression
