The Seng Huang Lee-owned yacht, which had been fourth out of the Sydney Heads, retired at sea after suffering a broken bowsprit.
Scallywag was one of five super-maxis in the race, which started at 1 p.m. local (0200 GMT), with Black Jack holding about a minute lead out of the harbour.
The Mark Bradford-skippered Black Jack took advantage of a drop and shift in the breeze as the leading yachts approached the entrance to the harbour.
'Bad thoughts drive me on' - Team GB sailor McIntyre
However, by shortly after 5 p.m. they were in fourth place, having being overtaken by fellow super-maxis Comanche, Wild Oats XI and Infotrack.
Last year's winner Comanche had headed further out to sea and were sailing at more than 27 knots with about 540 nautical miles until the finish in Hobart, while the three other big boats were closer to shore.
Scallywag was not the only withdrawal with the Gordon Ketelby owned and skippered Zen having to pull out with broken rigging.
Comanche took line honours last year in a race record of one day, nine hours 15 minutes and 24 seconds after winning a protest against Wild Oats XI for an incident shortly after the start on Sydney Harbour.
Wild Oats XI actually crossed the line first at Constitution Dock in one day, eight hours, 48 minutes and 50 seconds but had a one-hour penalty imposed following a near collision with Comanche about 15 minutes after the start.
Wild Oats XI had snuck past Comanche with less than 10 miles remaining as both boats were becalmed in fickle winds on the River Derwent and finished about 26 minutes in front, but was then relegated after the protest.
The Oatley Family-owned Wild Oats XI have won eight line honours titles and are considered the favourites because they had beaten Comanche across the line last year.
A total of 85 yachts entered the 630-nautical mile (1,170 kilometre) race, which is expected to be sailed in light to moderate north-easterly winds of between 15-25 knots and cause little in the way of dangerous conditions.
The forecast will be welcome news for organisers who will on Thursday commemorate the 20th anniversary of the 1998 race in which a sudden storm sank five boats, killed six sailors and resulted in 55 others being rescued.
The 2018 race will feature its first all-professional female crew on Wild Oats X, led by skipper Stacey Jackson. Australia's former Foreign Minister Julie Bishop was on the yacht at the start as a team ambassador but jumped off as it passed Bondi Beach.
Refreshed Scott back on Finn duty at European Championships
Bouwmeester wins gold in women's laser radial in 2016