England lost 2-1 to the Swedes after two early goals from Kosovare Asllani and Sofia Jakobsson, with Frakn Kirby's consolation goal proving to be in vain.
However, Neville says he is not overly disappointed to have lost the match as he was targeting the win.
‘Players are not robots’ - England’s Wiegman against biennial World Cup
"We came here to win it and not finish fourth," said Neville.
"Well done to Sweden but it is a nonsense game. What it leaves us is that we have another 15-20% to go. This is sport. We have to come back in four years and be better."
Neville sparked outrage with his comments, with former goalkeeper Siobhan Chamberlain, goalkeeper as England finished third in the last World Cup, saying she was honoured to win the bronze last time out.
The final of the Women's World Cup takes place on Sunday.
Women's World Cup Qualification UEFA
England smash eight past North Macedonia in Wiegman's first game
Tokyo 2020 (W)
Sweden v USA - Olympics clash, as it happened