England lost 2-1 to the Swedes after two early goals from Kosovare Asllani and Sofia Jakobsson, with Frakn Kirby's consolation goal proving to be in vain.
However, Neville says he is not overly disappointed to have lost the match as he was targeting the win.
"We came here to win it and not finish fourth," said Neville.
"Well done to Sweden but it is a nonsense game. What it leaves us is that we have another 15-20% to go. This is sport. We have to come back in four years and be better."
Neville sparked outrage with his comments, with former goalkeeper Siobhan Chamberlain, goalkeeper as England finished third in the last World Cup, saying she was honoured to win the bronze last time out.
The final of the Women's World Cup takes place on Sunday.