Having beaten Italy earlier in the year for their first win over a Tier One country, it was a stunning result for Georgia, who will face Wales again in the World Cup next year.
"Our second win over a Tier One nation this year - we've proved a lot of people wrong," said captain Merab Sharikadze, who also voiced his frustration at his team's lack of opportunities.
"We want to continue this - we made history today. We're not the team people think we are. We’ve beaten world-class teams and World Rugby needs to pay more attention to us and a lot of people need to be thinking that something has to change."
Wales led 12-3 at halftime via two Jac Morgan tries midway through the half but the conversion was the last points they managed.
Georgia defended brilliantly, grew in confidence and strength as their big bench added real power, and, after an Alexander Todue try, won it with Matkava's testing kick.
It sparked wild scenes of celebration among the Georgians. "We feel so proud. I am proud of the team and all the Georgian people in rugby," said man-of-the-match scrumhalf Vasil Lobzhanidze
"We all believed we could win, from the first minute to the last. We've made history. We are so proud of ourselves."
Wales had the best of the early exchanges but struggled to run their possession into dangerous situations. Eventually, flanker Morgan peeled round the blindside from a lineout with a neat sidestep for the first try after 21 minutes.
Three minutes later he got a second and though the crowd settled back for more, it proved the high-water mark of their day.
Winger Josh Adams, back in the side after breaking his hand, thought he had a third after chasing his own kick from halfway but it was chalked off for a forward pass, leaving Wales 12-3 up at the break.
The third quarter followed a similar pattern to the first as Wales had much of the possession but looked unstructured and rarely entered the 22 in the face of some committed Georgian defence.
They then lost winger Alex Cuthbert to the sin bin after a dangerous aerial challenge and Georgia took full advantage, building up some pressure before Tedo Abzhandadze kicked into acres of space for Todue to collect and score after an hour.
Abzhandadze added the conversion to his earlier penalty to bring it back to 12-10 but then missed with another kickable penalty as the Georgians enjoyed a long spell of dominance.
Wales finally roused themselves but a third try for Morgan was ruled out for a knock-on and it was the visitors who surged again to earn the all-important scrum penalty.
“We hoped to put more tempo into the game but all credit to Georgia, they did well in the set piece and came out on top," said Wales captain Justin Tipuric.
"It is a blow, we can't hide it, especially with Georgia coming up in the World Cup, but we just have to bounce back against Australia next week."
