Alpine ended last year fourth but are now sixth with French drivers Esteban Ocon and Pierre Gasly scoring just 14 points from five races.
To rub salt in the wounds, Aston Martin have gone from seventh in 2022 to second with 102 points and former Alpine driver Fernando Alonso is revitalised with four third-place finishes.
"The team managed to get fourth (in 2022). They have the means to get fourth, more so than others. I want them to be fourth. If they don’t, it’s going to be a failure," Rossi told the official website.
"If they fail by giving 500% best and turning this ship around, there will be extenuating circumstances and it bodes well for the future.
"If not, it’s the rule of business, there’s going to be consequences. And I won’t wait until the end of the year. The trajectory is not good. We need to fix the mindset of the team ASAP," added the Frenchman.
Rossi made clear Alpine team principal Otmar Szafnauer, who joined in February 2022 from Aston Martin, was in the hot seat.
"He is responsible for the performance of the team. That's his job," said Rossi. "There is no hiding here... the buck stops with Otmar."
Rossi said Ocon and Gasly were "doing their share of the job" and there was 'zero risk' of Renault walking away.
Alpine started the season with a target of defending fourth place and closing the huge gap to Red Bull, Ferrari and Mercedes.
Champions Red Bull have already scored more points than Alpine managed in all of last season while third-placed Mercedes and fourth-placed Ferrari are also a long way down the road.
McLaren, who have former Alpine reserve Oscar Piastri in their line-up after the Australian rejected the French team, are fifth.
"We started the season behind development targets," said Rossi. "We were lacking performance compared to where we wanted to be to cement P4. We have made a lot of mistakes, too many mistakes, over the weekend.
"When you compound that relatively lower performance and lack of operational excellence you end up in a difficult position."
Ocon had three time penalties in the Bahrain season-opener, the pair collided in Australia and in Azerbaijan suffered a catalogue of problems including Gasly's car catching fire in practice and then crashing in qualifying.
