Kuss began the race very much in the shadow of his more illustrious Jumbo-Visma teammates Jonas Vingegaard and Primoz Roglic but ended it with the red jersey on his shoulders.
He ended 17 seconds ahead of Denmark's Tour de France winner Vingegaard with Slovenia's Giro d'Italia winner Roglic one minute and eight seconds behind in third place.
It is the first time that a single team has won all three Grand Tours in a season.
Kuss's first Grand Tour overall victory was effectively wrapped up on Saturday and there was a celebratory mood amongst the Dutch team on the 101km 21st stage that concluded on a street circuit around the iconic landmarks of Madrid.
But the battle for the stage victory was a thriller as a small lead group featuring Groves and Remco Evenepoel was gradually reeled in by the peloton in the final 500m.
Belgium's Evenepoel (Soudal Quick-Step) eventually went off in search of his fourth stage win of the race but he could sustain his effort and was overtaken by Groves with 200m left.
Groves (Alpecin-Deceuninck) then poured on the power to take the win ahead of Ineos Grenadiers rider Filippo Ganna and Germany's Nico Denz (Bora Hansgrohe).
While Kuss celebrated the overall victory, Groves secured the green jersey for the points category - the first Australian to achieve that in the Vuelta.
Evenepoel, whose hopes of retaining his overall title evaporated on stage 13, ended with the consolation of the climber's jersey after a brilliant tour.
Kuss proved an unexpected but popular winner.
"It's incredible, I think today was the stage I suffered most in the whole race so now I'm just glad it's over," Kuss, who joined the team in 2018, told Eurosport.
"It's life-changing for sure but I'll still be me. I'll look back on this experience with a lot of fond memories. It's all just still sinking in really. It will take quite some time."
When Kuss took the red jersey from Lenny Martinez (Groupama-FDJ) on stage eight, most thought he was just keeping it warm for either Vingegaard or Roglic.
But when he managed to limit his losses on the stage 10 time trial and hang on to the red jersey his challenge grew.
When he then refused to crack in several ferocious mountain stages, one of which ended Evenepoel's challenge, he was left battling with his own teammates.
His advantage was whittled down when Vingegaard attacked him on a summit finish on stage 16 to take back more than a minute but he refused to bow in the closing days.
Jumbo-Visma sporting director Marc Reef said Kuss's victory highlighted the depth of the team.
"This is a victory for the whole team," he said.
"We are super proud. Sepp is just himself all the time. This means as much as Jonas's and Primoz's victory.
"It's special that he did all the three Grand Tours this year and was always an important guy in all our victories."
