The 23-year-old American was testing for the Toro Rosso Formula One team at Jerez when he crashed heavily on Thursday.
And after complaining of sore hands on Friday, doctors advised him not to compete in this weekend's end-of-year event at the Stade de France.
"We got confirmation of Scott's withdrawal this morning (Friday) immediately after he was given medical advice not to compete," said event organiser Fredrik Johnsson.
Race of Champions 2006
Ekstrom wins Race of Champions
16/12/2006 AT 19:47
Johnsson confirmed that a number of drivers have already been contacted with a view to partnering US rally champion Travis Pastrana at Team America in the ‘Nations Cup' section of the event.
His shortlist includes 2004 NASCAR Nextel Cup title-winner Kurt Busch and Robby Gordon, who has raced in NASCAR, the Grand-Am Sportscar series and even the Dakar Rally over the past 12 months.
Johnsson added: "We're now looking at different solutions for getting a team-mate for Travis Pastrana in the USA line-up but the fact that we're talking about drivers based on the other side of the Atlantic is not making our job any easier.
“We have already contacted Kurt Busch, as well as Robby Gordon who has often competed in The Race of Champions in the past. Hopefully we will get some news back very soon."
Speed's withdrawal is the latest blow for Team USA, as he was only announced this week as a replacement for current NASCAR champion Jimmie Johnson, who was forced out on Sunday with a broken wrist.
You can see the whole event from the Stade de France in LIVE streaming video from 16:00 CET on
Race of Champions 2006
Loeb cruises into RoC semi-finals
16/12/2006 AT 19:02
Race of Champions 2006
Finland win RoC Nations' Cup
16/12/2006 AT 16:33