Brazil, who have never won the Olympic gold, started with a highly rated front three of Neymar, new Manchester City signing Gabriel Jesus, and Gabriel Barbosa, the 19-year old Santos striker who is expected to head to Europe before the transfer window shuts.
But the little-known South Africans matched them until they had Mothobi Mvala sent off 10 minutes into the second half.
Brazil pinned the visitors down for the rest of the game but could not score.
"That's football," Barbosa told television cameras after the game. "Sometimes the ball doesn't go in the net."
Barcelona striker Neymar had two of Brazil's best chances, forcing keeper Itumeleng Khune to make diving saves to thwart two rasping drives from outside the box.
Brazil player Neymar (L) vies for the ball with South Africa player Tyroane Sandows during their Rio 2016 Olympic Games First Round Group A men's football match Brazil vs South Africa, at the Mane Garrincha Stadium in Brasilia on August 4, 2016
Image credit: AFP
Jesus smacked the ball off the post from two yards out when it looked easier to score. And Barbosa should have done better when he scuffed a shot with only the keeper to beat.
But a big crowd in the National stadium in Brasilia went home unhappy after the second game of the day ended goalless.
That result was not the only surprise as their great rivals Argentina lost 2-0 to Portugal. It was the South Americans' first Olympics reverse in 13 games. Argentina were unbeaten in winning gold during their last two appearances in 2004 and 2008.
Portugal players celebrate a goal
Image credit: Reuters
Holders Mexico drew 2-2 with Germany in Salvador, with all four goals coming in the second half.
Later in the day, South Korea put eight goals past debutants Fiji. Ryu Seungwoo got a hat trick in the demolition in Salvador, with seven of the eight goals coming in the second half.
In Thursday's final match, Nigeria - despite landing in Brazil just hours before kick-off - beat Japan 5-4 with Oghenekaro Etebo grabbing four goals.
Earlier on Thursday, Iraq and Denmark kicked the tournament off with a 0-0 draw in the same stadium.
The first goals in the competition came soon after in Rio, where Honduras beat Algeria 3-2 in an enthralling match.
The opening ceremony for the Rio Games takes place on Friday but the men's football tournament began 24 hours earlier, a day after the women's football teams began their quest for gold. Twelve women's teams are in Brazil, along with 16 men's sides.
The Olympic football competition is being played in host city Rio de Janeiro and six other cities.