NZR in March appointed Scott Robertson to take over the All Blacks after the World Cup against the wishes of Foster, who had urged the governing body to hold off on a coaching decision until after the tournament.
Foster also nearly lost his job last year after the All Blacks' first home series defeat to Ireland but two of his assistant coaches were fired instead following a NZR review.
Appointed after the 2019 World Cup on a two-year deal, Foster said NZR's interest in Robertson, the hugely successful Canterbury Crusaders coach, had made his position awkward from the start.
"When I got the job, it always seemed to be under a condition, didn’t it? It was the two-year contract," he told New Zealand radio station Newstalk ZB on Friday.
"They had another candidate that half the people wanted and from then on it was the lens that they looked (through) everything that we did.
"I wouldn't say there was a lack of trust, I just think when you think you've got a plan B in your pocket, sometimes you don't back plan A probably as much as you could."
With the help of a new forwards coach and former Ireland boss Joe Schmidt on his staff, Foster was able to turn the All Blacks around and they fell just short of a record fourth World Cup title, falling 12-11 to South Africa in the final.
"I’m at peace that we did everything we could, that we gave it everything we got," he said. "But still there’s always a massive disappointment we couldn’t get across the line."
Now unemployed, Foster said he had brushed off coaching offers in the leadup to the World Cup to focus on the All Blacks' campaign and a lot of opportunities were now gone.
Australia, though, are looking for a new coach after the resignation of Eddie Jones.
Having won back-to-back World Cups as an All Blacks assistant coach in 2011-15, Foster said it would be hard to move on to another nation.
"It’s immensely difficult to talk about trying to coach another country when you’ve just had 12 years with the best team in the world and who are so close to my heart," he added.
"Quite frankly I just need to breathe a little bit before I go down that path."
