"Our approach and that of Joern Andersen did not agree anymore as the coach was developing in a different direction," club manager Christian Heidel told reporters.
The club also sacked assistant coach Juergen Kramny and appointed under-19 coach Thomas Tuchel as Andersen's replacement.
Andersen joined Mainz in 2008, steering them to promotion in his first season in charge.
"I regret the decision of the club but our visions were different," Andersen told reporters.
Mainz lost 2-1 to lowly Luebeck and were eliminated in the first round of the Cup.
Thank you for reading 5 articles this month* Join now for unlimited access
Enjoy your first month for just £1 / $1 / €1
*Read 5 free articles per month without a subscription
Join now for unlimited access
Try first month for just £1 / $1 / €1