Bruce Mouat's rink had won three consecutive contests to surge back into contention for a top-six spot but ran into an in-form Japanese quartet led by skip Yuta Matsumura in Lethbridge.
Team Matsumura raced into a 6-1 lead but the Scots responded as Mouat played a hit and stay on a Japanese stone to score three in the sixth end and reduce the deficit to 6-4.
That gap was then just one by the time of the tenth end but Matsumura hit out a Scottish stone with his final effort to score one and seal an 8-6 victory that guarantees a place in the knockouts.
Japanese lead Shinya Abe said: "We qualified for the play-offs? Oh, fantastic! I didn't know that.
"That's what we are here for, so I'm very happy. That was a tough battle, Scotland came back really well but it was a good win."
Defeat drops Scotland's record to 4-4 and they lie seventh in the standings, with one win less than sixth-placed Italy after Joel Retornaz's rink beat Korea 9-4 following a brilliant draw into the house to take four in the ninth end.
Meanwhile, reigning champions Sweden notched back-to-back victories to join Japan with a record of 8-1, as Russia were defeated 9-5 and Canada swept aside 9-4.
The final match of the session saw Germany pick up their second victory of the Championship as skip Marc Muskatewitz drew his last stone into the house to score three and take a 7-6 win over Netherlands.
That was the perfect response by the Germans to a 5-3 defeat to Norway earlier in the day – a session that also saw Olympic champions, the United States, down China 7-5 to keep their qualification hopes on track and in-form Switzerland thrash winless Korea 11-2 to move their record to seven wins and one loss.