The four-time world champions, who crashed out of last year's World Cup group stage after also losing to Japan, have won just four of their last 17 international matches, putting coach Hansi Flick's future in doubt.
Junya Ito silenced the home crowd in the 11th minute, turning in a low cross at the far post.
The hosts quickly bounced back, levelling in the 19th following a superb passing move and a left-footed finish from Leroy Sane.
But their joy was short-lived as the visitors struck again just three minutes later with forward Ayase Ueda wrong-footing keeper Marc-Andre ter Stegen after another move down the right wing.
Ter Stegen denied Japan another goal when Ueda pounced on a defensive error a little later to charge through only to see his shot saved.
Japan had two quick chances in succession two minutes after the restart but Ter Stegen kept Germany in the game, as he did in the 70th, first palming Takuma Asano's shot from close range wide and then stopping Kaoru Mitoma's rebound effort.
Asano made amends in the 90th, slotting in to seal their second straight win over Germany before they added another in stoppage time with Ao Tanaka's header as German fans booed and whistled.
