Gauff, the highest-ranked player left in the women's draw, failed to recover from a series of misjudged shots and double-faulted eight times.
It was an uncharacteristically poor showing by the American world number four, who won the Beijing title in 2024 and made the semi-finals last year.
The 22-year-old was playing a day after she said she had received an "insane" amount of online racist abuse following her win over Chinese teenager Sun Xinran in the last round.
A group of supporters in the largely empty stadium tried to encourage her into a comeback, but Mertens was undeterred.
The Belgian world number 19 slammed down an ace on her second match point to wrap up the win in 78 minutes.
"She's an incredible player of course, I knew it was going to be a very tough match," Mertens said.
The 30-year-old dedicated her win to her coach, who is also her boyfriend.
Gauff spoke out on social media to condemn racist comments about her following her match with Sun, in which the American requested a video review over flickering LED lights on the stadium's advertising hoardings.
The request prompted some boos from the crowd. The umpire agreed to have the point replayed, and Gauff went on to win 7-5, 6-1.
Early on Tuesday, Gauff, who is Black, wrote on X: "Hey everyone, you can dislike me or disagree about what happened without being racist.
"The amount of racist comments and messages I've received in the last few hours is insane."
Responding to a post that contained a racial slur, Gauff clarified that she challenged the point before Sun's return when she noticed the lights flicker a second time.
Mertens will face Swiatek in the next round after the Pole swept aside American teenager Iva Jovic 6-1, 6-3 in one hour and 35 minutes.
"I had it under control from the beginning," said Swiatek. "I feel like I'm improving every day."
China's Zheng Qinwen is due to face 118th-ranked Alina Charaeva later on Wednesday.
Ukraine's Elina Svitolina, the sixth seed, will play American Ann Li.
