Klopp stunned Premier League leaders Liverpool on Friday when he revealed he will step down once the current campaign is over.
The Reds boss has been Guardiola's main competitor for domestic silverware during their time in England.
While Guardiola has largely come out on top in the trophy stakes during City's dynastic era, the Spaniard is well aware of the threat posed by Klopp and his team.
Paying tribute to the German after City's 1-0 win at Tottenham in the FA Cup fourth round on Friday, Guardiola said: "He's an absolutely incredible manager, I don't know him closely but I think he's an incredible person as well.
"I have the feeling that when he leaves at the end of the season, part of us at Manchester City is leaving too. They have been our biggest rival, Liverpool in the last years."
Klopp said he was leaving Liverpool because he no longer has the energy required to manage such a high-profile club after nine years at Anfield.
Although the two have never been close friends during their spell in England, their mutual respect is clear.
Guardiola can empathise with the 56-year-old over his decision after he walked out on Barcelona for similar reasons in 2012.
"All the managers will (feel tired) when you've done many years, I feel it completely. I felt it at Barcelona so I understand it completely," he said.
Although City have won the last three Premier League titles, their reign is under threat from Klopp again this term.
Liverpool currently sit five points clear of second-placed City, who have a game in hand.
The prospect of eventually playing a Liverpool team not managed by Klopp - one of the few coaches who can match Guardiola's intense competitiveness - is a pleasant thought for the City boss.
Their last showdown in the Premier League will come on March 9th when City visit Anfield for a crucial clash in the title race.
"He's been my biggest rival so I think he will be missed," he said.
"I am a little bit pleased because without him I will sleep a little bit better the night before we play against Liverpool but I wish him all the best. He doesn't admit but he will be back."
