Germany led through Ilkay Gundogan's first-half penalty and dominated thereafter, missing countless chances.
But a raft of second-half changes from Japan manager Hajime Moriyasu changed the impetus of the match, with the underdogs coming close to an equaliser before Ritsu Doan did bring his team level in the 75th minute.
Fellow substitute Takuma Asano fired home from a tight angle eight minutes later to send the crowd wild and secure Japan's greatest ever World Cup victory.
Germany, who finished bottom of their group in 2018, have now lost their opening match at successive World Cups and face a mammoth task to qualify for the knockout stage.
Hansi Flick's team face Spain in their second Group E game, while Japan will take on Costa Rica.
Fast Results
Scores
Racecards 
