The Brazilian forward will stay at Rayo until June 30 but the Madrid-based club do not have an option to purchase him on a permanent deal.
Costa made 28 La Liga appearances for Atleti last season, scoring six times, but has seen his chances restricted in the current campaign.
His move across Madrid represents a chance for the 23-year-old to enjoy regular first-team football before returning to Vicente Calderon.
Atletico Madrid to buy Porto's Colombia striker Jackson Martinez - agent
Costa first joined Atleti in 2007 and, following three loan spells and a permanent move to Real Valladolid, returned to the club in 2010.
Rafael Benitez: No rift at Real Madrid and Sergio Ramos is part of my plans