McLennan has been coming under increasing pressure since Australia crashed out of the pool stage of the World Cup for the first time in October under the leadership of coach Eddie Jones.
"We do not believe Mr McLennan has been acting in the best interests of our game," the unions, who represent every Australian state and territory barring New South Wales and Victoria, said in the letter.
"We no longer have any trust or faith in his leadership, or the direction in which he is taking rugby in Australia."
The hiring of Jones and sacking of Dave Rennie in January was driven through by McLennan but backfired spectacularly when the former England coach selected an inexperienced squad for the tournament.
The letter said McLennan had exceeded the power of his position by "exerting an undue influence on the operations and executives of Rugby Australia".
It also criticised McLennan's much-trumpeted strategy of trying to poach playing talent from the wealthier and more popular rugby league code, which RA chief executive Phil Waugh this week said would cease.
Jones resigned at the end of October saying the structural changes to the game required to make the Wallabies competitive were unlikely to be realised in the short-term.
Waugh launched a plan for structural reforms in August and this week announced that the New South Wales Rugby Union would be the first to hand over their high performance programme to centralised RA control.
Other states, most notably the powerful Queensland and Australian Capital Territory (ACT) unions, had qualms about the wholesale takeover of their high performance units, which includes their Super Rugby franchises.
Both of those unions were signatories to the letter and McLennan suggested that restructuring of the game was behind the attempt to oust him, which he vowed to fight.
"This will be the defining moment for the battle of rugby. It's all about money and control and we have been failing for years. We live in interesting times," McLennan told the Sydney Morning Herald.
"This is about principles. They are actually not putting the game first and it's about self-interest and parochialism."
