Colombia conceded a penalty and had a man sent off inside three minutes, with Carlos Sanchez dismissed for handling Shinji Kagawa's shot at a goal left exposed by terrible defending.
It was the first red card shown at this World Cup, and the second fastest in history at two minutes 56 seconds.
Japan should have increased their lead soon after as first Takashi Inui and then Yuya Osako wasted fine openings.
They were punished when Juan Quintero's clever free-kick, fired low under the jumping wall, brought Jose Pekerman's side on level terms six minutes before the break.
Colombia had never before come back to win a World Cup match in which they had conceded the opening goal - across 10 games - and star player Rodriguez was brought on in the second half, the Bayern Munich attacking midfielder having been deemed not fit enough to start.
But instead it was Japan who regained the lead, Osako finishing with a header from Keisuke Honda's corner.
Rodriguez had been struggling to leave a mark on the game, with Japan at last making their extra man count and dominating possession as might be expected, but there would be one big chance late on.
Freed into space after a clever backheel, Rodriguez's blistering strike from inside the box was deflected over the bar by a brave lunge from Osako.
This was the first time Japan had won a World Cup game on European soil, and the first time they had beaten a side from South America at the tournament.
For Colombia, a team who reached the quarter-finals in 2014, this was a miserable defeat that leaves them with two very important games against Poland and Senegal to come.