Rovers, one of four Super League sides to join the competition at the fifth-round stage, beat Wigan, Catalans and Warrington on their way to Wembley in 2015 but Championship side Oldham ensured they fell at the first hurdle this year.
The Roughyeds, in their first campaign back in the second tier, were 24-6 up at half-time courtesy of tries from Will Hope, Richard Lepori, Steve Roper and Adam Clay.
Shaun Lunt gave a strong Hull KR side hope early in the second half only for Gary Middlehurst to cross to make it 30-10 with 24 minutes to play.
Rugby League
Rugby World Cup postponed until 2022 after New Zealand and Australia withdrawals
04/08/2021 AT 19:50
James Webster's men reduced the deficit to eight points heading into the final 10 minutes through tries from Maurice Blair, Joe Wardill and Dane Tilse, his second, but Lepori completed one of the biggest upsets in recent years in the dying seconds.
Rugby League
Rudge: St Helens did rugby league 'proud' by winning maiden Challenge Cup title
05/06/2021 AT 15:39
Rugby League
Responsibility on Challenge Cup final to "showcase women's game at its best"
04/06/2021 AT 12:59