The Dutchman began Saturday's stage three at Storheia top of the general classification but ended up giving up the lead as he fell behind other challengers.
He had won the opening stage with an impressive late sprint but after four months from competing on roads he admitted he may not be able to sustain a challenge for the whole tour. However, after his second-place finish in Friday's sprint event, to see him fall back so emphatically was a surprise.
The 24-year-old Dutch rider finished in 23rd on the day, with a cold preventing him from competing at the peak of his ability.
'Sagan playing bumper cars' – How the big-name sprinters fared on Stage 2
Van der Poel explained: "I was not completely broken, but I got a bit sick last night.
Stage 3 finish: Odd Christian Eiking takes victory for Norway at top of Storheia
"I felt pretty bad on the bike all day. I tried, but I immediately felt there was not much I could do about it.
"My nose was blocked and I had a headache. Above all I had a lot of problems with my head and felt a bit sick."
His challenge suffered noticeable towards the end of the stage, when the last kilometre's steep ascent compromised his endurance and his placing slipped down the peloton. Warren Barguil, Alexey Lutsenko and Odd Christian Eiking - the stage winner - impressed in his place.
"I told the team at the start of the day that I didn't feel very well, but we tried it anyway, because I was the only one up there on GC," Van der Poel said of his difficulties.
WATCH: Van der Poel left surprised as Coquard takes Arctic Race win
"But I knew immediately that it wasn't going to be. I'm not saying that if I was completely fit I would have been able to go with the best guys right to the end, but when I saw my power numbers, I normally should have been able to be there for much longer."
Barguil now leads the general classification, and while Van der Poel has not given up hope, he admitted it may not even be within his powers to compete at all on Sunday.
"I hope it will be a little better by tomorrow. If I'll feel well enough to start tomorrow, I don't know yet."
Recap: Which teams 'won' the spring classics campaign and which must do better...
Van der Breggen, Van Aert & Roglic - The top 10 riders of the 2021 season so far