Jeffrey Schlupp and Riyad Mahrez looked to have put Leicester in control of proceedings as they scored either side of Michael Kightly's 39th-minute strike.
However, Leicester were unable to make their first-half dominance pay and eventually Burnley found their range as Ross Wallace netted a free-kick in the sixth minute of stoppage time.
"We should have won the game, we did enough to win the game," he told a post-match media conference. "But in the end we conceded in the 96th minute.
"I'm annoyed today, not disappointed, I'm annoyed.
"We should win the game, I think we're better than we were last year and I don't think they are as good as they were last year.
"They'll be pleased because they've got a result with all the injuries that they've got, but ultimately we should have won the game today.
"We've over-achieved in some games but today we've not capitalised on the situation we've found in front of ourselves. We've been in front twice, we should have won the game."
