Dean Elgar (35) was axed as captain, although he remains a member of the squad which starts a series against the West Indies on February 28th.
Chief selector Victor Mpitsang was another casualty of the shake-up, with CSA stating he had been "released" from the role.
Bavuma became the first black African to captain South Africa in cricket when he was named leader of the one-day international and Twenty20 international teams in March 2021.
At the same time, Elgar was named Test captain.
Elgar’s stint as captain started well with Test series wins against West Indies, India and Bangladesh and a shared series away against world champions New Zealand.
But they were heavily beaten in their two most recent series in England and Australia.
Bavuma, meanwhile, came under criticism for his poor batting form in T20 internationals, notably in the T20 World Cup in Australia last year.
According to Friday's statement, Bavuma will remain captain of the one-day team but will relinquish his role in the T20 side.
The announcements were made by Enoch Nkwe, CSA's director of cricket, who said it had been decided to review the selection system. Mpitsang was relieved of his job.
As an interim measure, the new national team coaches, Shukri Conrad (Test) and Rob Walter (limited overs) will play a leading role in selection.
"Temba is a player that has vast captaincy experience both at domestic level and on the international stage," said Nkwe.
"We trust he will deliver on all our expectations and help carry the team forward after some excellent work by his predecessor Dean during the same period.
"At the same time, I would like to sincerely thank Dean for all his commitment to the role over the past two years.
"He helped the team navigate through some stormy waters and put them in a good position on the ICC World Test Championship table."
Nkwe named a 15-man squad for the two Tests against West Indies.
Five players who played in the losing series in Australia in December and January – batsmen Sarel Erwee (33), Rassie van der Dussen (34) and Khaya Zondo (32), wicketkeeper Kyle Verreynne (25) and fast bowler Lungi Ngidi (26) – have been dropped while Theunis de Bruyn (30) has retired.
Uncapped Tony de Zorzi (25) is likely to make his debut as an opening batsman, while Keegan Petersen (29) and Ryan Rickelton (26) return after missing the tour of Australia because of injury.
Batsman Aiden Markram (28) and all-rounder Wiaan Mulder (24) are recalled as is spinning all-rounder Senuran Muthusamy (28) whose only two Tests were in India in 2019.
Rickelton’s omission from the Australia tour led to criticism of Mpitsang, who said he was left out because of concerns over an ankle injury despite the player saying he was fit - and then scoring heavily in domestic cricket while the tour was taking place.
