Cavendish, who will retire at the end of the year, fell off his bike with 64km left in the eighth stage won by Pedersen, and was taken into an ambulance for checks before his withdrawal was made official by race organisers.
The Astana-Qazaqstan rider was looking to become the only man with 35 stage wins on the world's greatest cycling race, to beat the record he shares with Belgian great Eddy Merckx, who bagged 34 victories from 1969-75.
On Friday, Cavendish came close to achieving his goal when he took second place in the seventh stage in Bordeaux after suffering a mechanical problem in the final straight.
Cavendish held his right arm after crashing on Saturday, lying down in agony, his face a mask of pain and blood on his elbow. He then held his head in disbelief as he climbed into the race ambulance where doctors strapped up his shoulder.
"Mark Cavendish broke his right collarbone. Moreover because of the fracture, an ostesynthesis screw in the acromioclavicular joint (shoulder) is loose," the team said in a statement.
The screw has been in his shoulder since a crash in the 2017 Tour de France.
His withdrawal was effectively confirmed a few minutes later when the ambulance's door was slammed shut, bringing a close to a remarkable Tour de France adventure that started in 2007.
"Second yesterday, and today have this happen to Mark it's hard because we know his shape's here, we know he has the legs," Mark Renshaw, who was Cavendish's lead-out man from 2009-2011 and in 2016 and joined Astana-Qazaqstan as a sprint adviser, told reporters.
"I won't lie, I cried. Everyone in the team is hurting."
Cavendish was taken to a hospital in Perigueux, a team representative told Reuters.
"It is an emotional day," Tour director Christian Prudhomme said. "He's the best sprinter in the history of the Tour de France. During two or three seconds yesterday we thought he would succeed in reaching his goal, and today, it's over. We are sad, the Tour de France is sad.
"Mark deserves the respect of the Tour. He always will be welcome with or without his bike."
Cavendish, who started as a self-described "boy who wanted to fight the world", won his first stage in Chateauroux in 2008, bursting into the limelight as he claimed another three victories that year.
It was the first of nine Tours in which Cavendish bagged at least a stage win.
LOVE FOR THE TOUR
Cavendish has repeatedly proclaimed his love for a race that gave him the best moments of his career with two green jerseys for the winner of the points classification, but also the most heartbreaking one on Saturday.
While he was beaten three times in bunch sprints by Belgian Jasper Philipsen, Cavendish clocked the fastest speed in two of them, showing his 14th and last Tour would not just be an easy farewell ride.
It ended in the most brutal fashion in a crash that nobody saw coming in the haze of a sunny afternoon before the expected final sprint on a mostly flat stage.
Cavendish's withdrawal comes on the day Philipsen was beaten for the first time in a sprint this year as the Belgian took second place behind former world champion Pedersen.
Belgian Wout van Aert finished third.
Defending champion Vingegaard retained the overall leader's yellow jersey with a 25-second advantage over Tadej Pogacar with Australian Jai Hindley in third place, 1:34 off the pace.
Briton Simon Yates crashed some 5.5km from the finish line and lost 47 seconds, slipping down to sixth overall from fifth -- a spot now occupied by his twin brother Adam.
Sunday's ninth stage ending with the ascent to the Puy de Dome - for the first time since 1988 - will start from Saint Leonard de Noblat, where former France great Raymond Poulidor was buried in 2019.
Poulidor's grandson Mathieu van der Poel, who won a stage on the Tour and wore the yellow jersey in 2021 and has been one of the best classics riders in recent years, said it would be an "emotional" stage, having spent summer holidays in the small central France town.
