The team of Luis Fernando Suarez played most of the game on the back foot but won thanks to the stunning curling shot of right-back Keysher Fuller in the 81st minute.
After defeating Germany, Japan came into Ahmad bin Ali Stadium feeling confident, but they lacked the edge to get past a staunch Costa Rican defense.
As Fuller’s late goal, the game took a dramatic turn after it appeared to be destined for a dull tie.
The outcome leaves Group E very open. Amazingly, Costa Rica, who fell to Spain 0-7 in their opening match, is still in the running to get to the next round.
A win on Sunday would have sealed Japan’s spot in the round of 16, but they will now need to get a result in their last game against Spain to keep their hopes of qualification alive.
Japan made five changes to the team which shocked Germany in their opening match as coach Hajime Moriyasu made full use of his 26-man squad.
In temperatures of 32 degrees Celsius (90 degrees Fahrenheit) at Ahmad Bin Ali Stadium, Japan immediately went on the attack and won a corner within 30 seconds to signal their intent.
Costa Rica coach Luis Fernando Suarez had resisted making wholesale changes to the team thumped by Spain, dropping teenage winger Jewison Bennette and defender Carlos Martinez to the bench.
Gerson Torres and 34-year-old centre-back Kendall Waston took their place in an ageing XI which featured four players from the 2014 World Cup quarter-finals.
But Japan’s early promise melted in the Doha sun and the first shot at goal a tame effort which sailed high and wide did not come until 10 minutes from half-time through Costa Rica’s 30-year-old Joel Campbell.
A dreary first half came to a close with “Los Ticos” having had more of the ball but neither side mustering a shot on target.
Moriyasu had seen enough and made a double change at the break, one of them the introduction of Bundesliga-based striker Takuma Asano, who scored the winner against the Germans.
The Blue Samurai were immediately more cutting and midfielder Hidemasa Morita drew the first proper save of the match within seconds of the restart with a powerful strike.
Moriyasu threw on Brighton attacker Kaoru Mitoma for a defender just after the hour as Japan chased the victory that would put them on the cusp of the last 16.
It was all Japan as the match ticked into the last 20 minutes in front of 41,000 spectators.
Substitute Junya Ito looked to have wriggled free and was closing on goal, only for Costa Rica defender Francisco Calvo to cynically tug him back on the edge of the box, earning a yellow card that could easily have been red.
Costa Rica had done nothing for most of the game as an attacking force, but nine minutes from normal time they stole it when Fuller’s shot took a deflection and looped in.