According to media reports, first-team coach Michel Estevan could also be on the brink of dismissal.
"Duverne has left," club president Michel Salerno told Reuters by telephone.
"What worries me is what I saw at the Parc des Princes. Some players have no place at Arles-Avignon," he added, referring to Saturday's 4-0 hammering by Paris Saint-Germain.
Promoted Arles-Avignon are bottom of Ligue 1, having lost their first five games.
The south eastern city of Arles is better known for bullfighting than football and the club play their matches at nearby Avignon.
Duverne has only been fitness coach for two months and carried out the same role for France at the World Cup in June.
He was involved in a heated exchange in South Africa with captain Patrice Evra when the players boycotted a training session in support of striker Nicolas Anelka, who had been expelled from the squad.
France were knocked out in the first round.
Despite signing former Real Madrid defender Francisco Pavon and Greece striker Angelos Charisteas, Arles-Avignon have conceded 10 goals and scored only twice in the league this season and the heat is on Estevan.
The coach, who was fired and then reinstated during the close season, took over in 2005 and has lifted the club from the depths of division five.
‘Maybe I’ll be signed by Wrexham, who knows? English football would suit me – if an offer came, I wouldn’t think twice, I’d go immediately’ Euro 2024 cult hero reveals his UK dream
‘Managing Leeds? It was an option that appeared, but it wasn’t the right timing. I decided it wasn’t a good idea to leave the club I was at mid-season’: Premier League boss admits to turning down opportunity to replace Jesse Marsch in 2023
