The 23-year-old moved to White Hart Lane in January 2013 but has found his first-team opportunities limited at Spurs.
Having made just six starts in the Premier League this season, the German has been linked with a return to the Veltins Arena.
Despite Spurs manager Tim Sherwood hinting that Holtby could be allowed to leave, Keller said it was unlikely Schalke could finance such a deal.
"We can't really imagine such a transfer financially," he told Bild.
"Talks have not yet reached an advanced stage."
Keller did add that Holtby would represent a boost for his squad, with midfielder Jermaine Jones set to leave the club in the current transfer window.
"Lewis has already played as a defensive midfielder under me," Keller added. "He can play there and is tactically very well-schooled."
Holtby originally moved to Schalke from Alemannia Aachen before being loaned to Bochum and Mainz during his time at the club.
Get the best features, fun and footballing frolics straight to your inbox every week.
Thank you for signing up to Four Four Two. You will receive a verification email shortly.
There was a problem. Please refresh the page and try again.