Spaniard Nadal needed to win the tournament to take the rankings top spot from compatriot Alcaraz, but straight-set losses to Fritz on Sunday and Felix Auger-Aliassime (22) earlier on Tuesday left him needing Ruud to lose in straight sets to keep his hopes alive.
The Norwegian, however, won the first set 6-3 against Fritz, capitalising on his much improved serve.
The American did fight back in the second set to win it 6-4, setting up a deciding third set.
Ruud and Fritz went with serve in the third to set up a tense tie-break which the Norwegian held his nerve to win the tie.
Alcaraz, who is not competing at the tournament due to injury, will be confirmed as the youngest year-end number one since the ATP rankings began in 1973.
With his 6-3 6-4 loss to Canada's Auger-Aliassime, the record 22-times Grand Slam champion Nadal has now lost four successive ATP Tour matches for the first time since 2009.
Auger-Aliassime, now coached by Nadal's uncle and long-time mentor Toni who sat in his box, will face Fritz in his final group match with a chance of making the last four.
"I had one match to get used to the conditions and I felt my game was better today, I was hitting it with precision and consistency," Auger-Aliassime said.
It was a disappointing display by Nadal who looked rusty having played only one tournament since the US Open.
"There are a couple of positive things. I was able to play two tournaments in the past three weeks. That's a positive, something that I was not able to do for a while," he said.
"I don't think I forgot how to play tennis, how to be strong enough mentally. I just need to recover all these positive feelings and all this confidence and all this strong mentality that I need to be at the level that I want to be.
"I don't know if I going to reach that level again. But what I don't have any doubt is that I (am) going to die for it."
