Liverpool famously have previous when it comes to shootouts at the Ataturk Stadium, having won the Champions League on penalties at the same ground a decade ago.
But their luck was out on this occasion, as Lovren blazed over from 12 yards with the last kick before sudden death.
Penalties arrived after a tense 120 minutes in Istanbul, with Tolgay Arslan's fine strike 18 minutes from the end of normal time cancelling out Mario Balotelli's first leg goal to level the tie at 1-1.
Demba Ba - Liverpool's nemesis in last season's doomed Premier League title bid - should have sent Besiktas through in stoppage time, but the former Chelsea man crashed against the crossbar from six yards.
There were few signs of nerves through the shootout as the first nine kicks were netted, but Lovren's wild effort ensured lightning would not strike twice for Liverpool 10 years on from their dramatic victory over Milan.
Besiktas were subdued in the early going, seemingly inhibited - rather than inspired - by the fervent atmosphere created by their fans as Liverpool comfortably kept their first-leg lead intact.
Slaven Bilic's side began the match on the front foot, but the first attempt on goal came from Liverpool's Raheem Sterling 20 minutes in.
Alberto Moreno latched on to a clever flick from Balotelli - making his first start since November - down the left before pulling back to Sterling, whose effort from 20 yards was palmed away by Cenk Gonen.
Daniel Sturridge and Moreno then tested Gonen in quick succession just after the half-hour mark as Liverpool warmed to their task and quietened the vociferous Besiktas faithful.
But Besiktas' players and fans were re-energised by the interval, and Bilic's men got their just reward for a much-improved display when substitute Arslan curled a superb left-footed shot beyond Simon Mignolet after good work from Gokhan Tore and Ba.
The hosts continued pressing, but Liverpool just about held firm to force extra time - thanks to Ba missing a golden chance to settle matters inside 90 minutes.
Besiktas remained on top through the extra period, Ba again going close only for Mignolet to stand firm at his near post.
However, the Senegal striker's misses would not to prove costly, as Lovren followed up a string of well-placed efforts by launching the 10th penalty of the ensuing shootout high and wide.
‘After Manchester City’s recent form, maybe they’re the underdogs against Manchester United!’ Former Red Devils defender on this weekend’s derby
‘Arteta, Alonso, Emery, me… none of us were physical players – we needed the understanding of the game. That probably helped us move into management’: Premier League boss reveals reasons for natural career progression
