Pogacar, who attacked several times in the first two hilly stages, lost more than a minute to Dane Vingegaard as his lack of fitness showed in the fifth stage on Wednesday after his preparations were hampered by a fractured wrist in late April.
Vingegaard's brutal acceleration near the top of the Col de Marie Blanque was too much for Pogacar, who had to dig deep to limit the damage.
Australian Jai Hindley (27) won the stage and took the overall leader's yellow jersey from Pogacar's UAE Emirates team mate Adam Yates (30), with Vingegaard moving up to second, 53 seconds ahead of sixth-placed Pogacar.
The Slovenian, who is often seen with a smile on his face regardless of the circumstances, looked stunned at the finish line after he and his team were caught by surprise.
"Tadej, what do we do?," one of his teammates said on the team radio as a group of 36 riders, including Hindley, broke away early in the stage.
UAE Emirates controlled the pace but Pogacar was without a teammate when Vingegaard made his move.
"It's a big blow, it's the first mountain stage," Pogacar said.
With a tougher mountain trek through the Col d'Aspin and Col du Tourmalet on Thursday, the two-time Tour champion will be worried that he might get hit again.
Damage control and hoping to peak in the final week in the Alps seem to be Pogacar's best hope of regaining the title he lost to Vingegaard last year.
"Vingegaard was very, very impressive and very good game by Hindley... In the first two stages near Bilbao on the short climbs, Tadej was fine but today it's a bit different," UAE Emirates team manager Mauro Gianetti told reporters.
"Fortunately we had an advantage from the first two stages (after Pogacar picked up bonus seconds). We lost time but we're still in the game
"Vingegaard is super good but Tadej was not good enough to be with Jonas. Hindley won the Giro d'Italia, we had said that the battle is not only between Tadej and Vingegaard and we have to expect other contenders and we see that Hindley is a big contender."
Pogacar was focusing on Vingegaard, but when it was time to react, the Slovenian's head dropped.
"We were hoping he would be in good condition but we knew since Liege-Bastogne-Liege (where Pogacar broke his wrist) it would be difficult to be at 100%," said Gianetti.
"We fought in the first days and took yellow. We're just 50 (53) seconds behind Vingegaard and it's only the fifth stage. The Tour is still long."
Pogacar, however, hinted there was not much he could do for now.
"I hope we'll have better legs than Jonas in the last week," he said.
