The federation handed down the punishment following an extraordinary meeting late on Tuesday.
Seto, the reigning world champion in the men's 200 and 400m individual medley, resigned as captain of Japan's Olympic swim team last month after admitting to the affair.
"How can I apologise? I have been asking myself if I can, but I think my apology is to continue swimming, restoring the trust of my family, who have been deeply hurt by my irresponsible behaviour," sports daily Hochi Sports quoted him as saying.
"I would like to seriously face swimming with the determination to start over so that my family and everyone will recognise me as a swimmer again. I’m really sorry,” added Seto, who a bronze medal at the Rio Olympics in 2016.
The scandal has tarnished Seto's reputation and he has already agreed to sever ties with sponsor All Nippon Airways.
(Reporting by Jack Tarrant; Editing by Peter Rutherford )