Guillaume Martin has admitted that he "wouldn't put his hand in the fire" to vouch that the peloton was drug free.
The Cofidis rider ultimately finished in 11th after mounting an early tilt at the podium.
Former riders Christophe Bassons - a longstanding critic of doping in professional cycling - and Stephane Heulot raised their suspicions since the end of the Tour.
Bassons said that, "today there is a new obstacle with performances that seem abnormal."
- 'I'm the number one British cyclist, not you': When Tom Simpson conquered the world
- Quintana says he has nothing to hide as French police release two from custody
- Vervloesem and Moniquet to join Lotto Soudal
Heulot said in 2008 that current UAE Team Emirates manager Mauro Gianetti was "ingrained in doping" and said "he felt the urge to vomit" in regards to this years Tour.
Speaking to Le Parisien, Martin admitted that he understood that cycling's past gave critics good reason to continue to cast doubt on current riders.
"Every year, the winner of the Tour is suspect," he said.
"This is the price of the sport's troubled past. One has to live with it.
"I can't allow myself to have doubts and say to myself 'I'm 11th but he or he is suspicious'. Otherwise, I'll get too discouraged. Even though I'm not naïve: I wouldn't put my hand in the fire to say that the whole peloton is clean.
‘Roglic was beaten by a better man, there’s not much he could have done differently’ – Wiggins
"Afterwards I see on social networks that some people even doubt me, even though I know exactly how I work. I can't do anything about the doubts. If Tadej Pogacar is clean, it's terrible to be accused of that."
Martin, whose 11th place was his best result from four Tour participations so far, said that cycling had a problem with so many team managers and directeurs sportifs having worked during "darker moments" of the sport, but added that everyone has a right to a second chance.
"One of the problems of our sport is that many of its leaders were in the driving seat in the dark moments. This is embarrassing for credibility. But, at the same time, there is a second chance. We have the right to change and not be condemned for life," he said.
"There is no clear ethical answer [if I could work with them]. If I feel that the sincerity of the repentant, I would say yes, but it's a question of feeling.
"The Tour recovers from everything. It recovered from Covid-19 when we said that it wouldn't take place. I believe in human optimism – even a new 'dirty trick' wouldn't bring it down."