The German had been one of the favourites to battle world number one Naomi Osaka for the title in the Spanish capital this week and had a chance to overhaul the Japanese at the top of the rankings.
The ankle injury is another setback for 2018 Wimbledon champion Kerber, who missed two weeks of the claycourt season after picking up a virus in Monterrey last month, and she set her sights on returning in time for next week's Italian Open.
"I think it's the worst [clay court season] I ever had. I was really looking forward to this year," Kerber told a news conference.
Opinion: No multiple major winners again, but Barty’s 2021 season should be given more respect
"Hopefully I can be ready for the next big ones. I hope there are big chances to play in Rome."
Martic automatically qualified for the last 16 as a result of Kerber's withdrawal and will meet the winner of the last 32 tie between Donna Vekic and Kristyna Pliskova taking place later on Tuesday.
US Open
'She has a great future' - Kerber pays tribute to rising star Fernandez after defeat
06/09/2021 AT 06:32
US Open
Fernandez - Kerber - US Open Highlights
05/09/2021 AT 23:01