It did not look like India had enough on the board after they were bowled out for 213 in 49.1 overs, losing all 10 wickets to spinners for the first time ever.
Dunith Wellalage (5-40) wrecked India's top order while Charith Asalanka (4-18) mowed down the tail to restrict India to a below-par total.
Sri Lanka's batting, however, let them down and they managed 172 all out in 41.3 overs with only Wellalage, who made 42 not out, and Dhananjaya de Silva (41) offering some resistance.
The defeat snapped Sri Lanka's 13-match winning streak in one-day internationals and also means their clash against Pakistan on Thursday will effectively be a semi-final.
"It was a good game," India captain Rohit Sharma said after their victory.
"For us to play a game like that under pressure, it challenged a lot of aspects of our game. We definitely want to play on pitches like that to see what we can achieve."
Flying high after thumping Pakistan at the same R Premadasa Stadium a little over 12 hours ago, India were brought down to earth after Rohit elected to bat.
There was no early sign of the impending collapse though as Rohit (53) and Shubman Gill (19) forged an 80-run opening stand before Wellalage ripped the heart out of India's top order.
The left-arm spinner claimed three wickets in his first 13 balls, removing both the openers and Virat Kohli.
Rohit completed 10,000 ODI runs in his 241st innings, second fastest to the feat behind teammate Kohli, who reached the milestone in 205 innings.
Ishan Kishan (33) and KL Rahul (39) staged a brief recovery but Wellalage broke the 63-run stand to trigger another collapse.
Off-spinner Asalanka then ran through India's lower order and India reached the 200 mark thanks to Axar Patel's (26) defiant cameo.
The target was modest but Sri Lanka were soon grappling with a collapse of their own.
Jasprit Bumrah dealt the first two blows before the Indian spinners joined the party.
Kuldeep Yadav (4-43) struck in successive overs and Ravindra Jadeja removed home captain Dasun Shanaka to reduce Sri Lanka to 99-6.
Wellalage and de Silva combined in a 63-run partnership to revive Sri Lanka but once Jadeja separated them, Sri Lanka collapsed in a heap.
