Russian Khachanov, seeded 27th, fired an unreturnable serve on match point to seal the win over the Australian, who had knocked out world number one Daniil Medvedev in the previous round.
Khachanov was better in the intense match's biggest moments, saving seven of the nine break points he faced, often with his thundering serve.
"Crazy match like I was expecting it would be," he said in an on-court interview after his second straight five-set win.
"I am ready to run, to fight, to play five sets. We played again for almost four hours and that's the only way to beat Nick I think."
Kyrgios, known as much for his explosive temper as his blistering serve, came out with surprisingly little energy and dropped the first set when Khachanov hit a perfect lob.
The Wimbledon finalist, who has the most wins of any player on Tour since June, received medical attention on his left knee prior to the start of the second set, raising the prospect he might retire from the match when he said he "could not walk".
But he persevered and broke early in the second set, which he claimed with a crosscourt backhand winner as he began to grow more animated, yelling at the players' box for encouragement and bringing the crowd to its feet with some spectacular shotmaking.
Khachanov grabbed the third set and the players exchanged breaks early in the fourth to set up a tiebreaker dominated by Kyrgios.
Momentum swung firmly in Khachanov's direction when he broke to open the decider on a backhand error by Kyrgios and continued to hold serve until the finish.
"I tried to stay focused in the match," Khachanov said.
"I had some opportunities in the fourth set but he played an amazing tiebreak and what can I do? I can stay focused and try to win the next set," Khachanov said.
"I'm really proud of myself, I was really focused from the beginning to the end and I got the win."
Wimbledon finalist Kyrgios, who smashed rackets in frustration after the match ended, said he was "devastated" after the loss.
"I feel like I let so many people down," he told reporters.
"These four tournaments are the only ones that matter and I feel I have to start all over again."
Khachanov will next face Norway's Casper Ruud for a place in the US Open final.
