"Man, I made that so hard on myself," a relieved Ancer said after holing a four-foot putt to win the $4 million first prize in the Saudi-backed breakaway circuit event.
Ancer had started the day five clear of the pack on 15-under after bogey-free rounds of 63 on Friday and 62 on Saturday at the Hong Kong Golf Club.
But the Mexican saw his ultra-smooth swing of the first two days desert him in chilly and wet conditions on Sunday, as he returned a two-over 72 in front of big crowds, who turned out in force despite the deteriorating weather.
"The ball-striking wasn't there, but mentally I was really strong, so I felt really good," he said straight after the playoff, where he set up only his second birdie of the day with a superb approach.
"I just kept myself in it and hit the right shot at the right time there in the playoff."
Englishman Casey holed a spectacular bunker shot for a birdie at his final hole, the 16th, to complete a six-under round of 64 and draw level with Ancer and Australia's Smith (66) on 13-under par.
But he found a fairway bunker off the tee in the playoff to end his hopes.
"I played some great golf," said Casey, whose 'Crushers' won the team competition and a $3 million (£2.3m) first prize to go with his $1,875,000 (£1.45m) for a share of second place.
"I'm not going to measure it on the play-off hole, on one tee shot, something like that. It was a really good week."
For Smith, the 2022 British Open champion, it was the second agonising disappointment on Hong Kong's final green in four months.
In November, he saw New Zealand's Ben Campbell hole a birdie on 18 to pip him to the Hong Kong Open title.
Earlier, Ancer stumbled to his first bogey of the week at the par-three fifth after finding sand off the tee.
Another bogey at the short eighth reduced his lead to two strokes, though he repaired some of the damage with his first birdie of the round on the par-four 10th.
A disastrous drive into a stream on the 15th dropped him back to 13-under and Smith swept home a 20-foot birdie putt to draw level.
Double hole-in-one
US Masters champion Jon Rahm was LIV's biggest coup when he defected from the PGA Tour in December.
Rahm briefly got into contention at 12-under before two bogeys at the 14th and 16th ended his challenge as he finished 10-under par after a frustrating 71.
Chile's Joaquin Niemann won two of the previous LIV events this season, including last week in Saudi Arabia.
He sunk the week's first hole-in-one, at the 188-yard eighth, in a brilliant round of 63 to finish tied for fourth with another Mexican, Carlos Ortiz (66).
Finland's Kalle Samooja then aced the 149-yard second hole for the sixth LIV hole-in-one since the breakaway tour began in 2022.
Three-time PGA Tour winner Anthony Kim shot a final-round 65, his best round since making his first professional appearance since 2012 at last week's LIV Golf event in Jeddah.
"I just kept doing what I've been working on the last two months," he said.
"Obviously being away from the game for so long, it's been tough to practise and get all the things that I need to get prepared for the tournament, but I'm working on the right things at this moment."
