Wales had been on the brink of a 20th defeat in their last 21 Tests, but ill-discipline from Japan allowed them to enter their opponents' 22 and win a penalty, which was converted by Evans.
Fly-half Dan Edwards, winger Louis Rees-Zammit and centre Nick Tompkins scored tries for the home side in a seesaw contest, but a 20-minute red card for winger Josh Adams on the stroke of halftime did not help their cause.
Japan had been well-beaten by South Africa and Ireland in their previous two Tests, but looked on course to win in Cardiff after tries by winger Kippei Ishida and number eight Faulua Makisi, but they could not hold on.
"Testament to his (Evans) mindset to come on and take that on and slot it," Wales number eight Olly Cracknell said.
"They (Japan) were really challenging, it was a really hard match. They are so fit and play a fast game. The pressure is constant."
Wales will stay 12th in the world rankings with the win, which could be vital with the 2027 Rugby World Cup draw next month and New Zealand and world champions South Africa to face in the next two weekends.
"We weren't focusing on that too much, we are just trying to build," Cracknell said.
"There were things we wanted to improve from last week (a 52-28 loss to Argentina), and so it is progress."
Japan had never won in Wales and were the better side, but three yellow cards meant they too played a significant portion of the game with a player less, at one stage down to 13 in the first half.
The teams went to half-time locked at 7-7, but Adams' illegal clear-out at the ruck meant Wales spent the first 20 minutes of the second half a man down.
Japan took advantage and camped in their half, working their way into a 23-21 lead as the minutes ticked away.
When Japan's replacement lock Harry Hockings made a dangerous tackle on Wales' Alex Mann, however, it was the opportunity the home side needed to set up a scoring opportunity.
They won a penalty, and Evans, only on the pitch for two minutes, sent the Principality Stadium into raptures.
