Lucy Bronze scored a last-gasp goal for the Lionesses, who briefly celebrated, believing it was enough to win Group A1 and move into the Nations League semi-finals, but the Netherlands' Damaris Egurrola netted twice in added time as they beat Belgium 4-0 to clinch top spot.
"I'm very, very disappointed. That's the biggest emotion right now," England coach Sarina Wiegman told reporters at Hampden Park.
Both countries finished with 12 points but the Dutch edged England on goal difference 8-7.
"Obviously devastated to lose out on such small margins," England's Beth Mead told the BBC.
"We got the goals later on, we thought we'd maybe got over the line, but unfortunately it wasn't meant to be," she added.
"It's a tough one, I think the girls have worked really hard, I thought we were incredible. Unfortunately, it was out of our hands."
England came from behind to beat the Netherlands 3-2 on Friday in a must-win game to set up Tuesday's thrilling finale.
Lauren James scored two in the first half less than two minutes apart, while Mead netted in her first start with the World Cup finalists in more than a year due to a knee injury.
Alex Greenwood and Fran Kirby also scored for the Lionesses who were ruthless in their quest for goals, retrieving the ball from the net with no celebration to run back to the centre circle after each one.
England keeper Mary Earps, who was "gutted" after conceding twice against the Netherlands on Friday, made a spectacular last-gasp save before Bronze added England's sixth with her head in the 93rd minute, pumping both fists in celebration.
Yet the celebration soon turned to sadness at Hampden Park.
"I said to the players: I don't know what to say because I'm very proud but we're not through and that's very disappointing," Wiegman said.
"If you win 6-0 that doesn't happen very often anymore in the women's game. Six goals is a lot. If the Netherlands also scored four goals then it's not enough."
The two finalists of the inaugural Nations League tournament - or top three if Olympic host France are one of the finalists - earn berths in the Paris Games.
Germany's 0-0 draw with Wales on Tuesday was enough for them to clinch top spot in their group and move on to the February 23rd-28th four-team knockout round.
France and Spain clinched top spot in their groups.
