Asbestos, which was once widely used in fireproofing and insulation, has been linked to several health hazards including mesothelioma and other types of cancer.
The Tokyo Metropolitan Government and Tokyo 2020 were not immediately reachable for comment. Much of Japan is closed for New Year holidays.
The Asahi report said the Tokyo government found the asbestos in fireproof material sprayed on part of the structure supporting the Tokyo Tatsumi International Swimming Center's roof.
While Japanese construction industry regulations state asbestos should be removed or covered up, the Tokyo government deemed the amount not to be large in scale, in a stable condition and not accessible so decided to leave it, Asahi said.
Organisers had since decided to take "emergency countermeasures", Asahi said without providing further details.
The report is the latest headache for Tokyo 2020 organisers, who have faced concerns over elevated levels of bacteria in the water in Tokyo bay and how athletes will cope with sweltering summer heat, which has led the marathon event to be moved to the far northern city of Sapporo.