If that was the case, you're right, it shouldn't have been a Red. My guess was that he was smiling at you the whole time? Awful ref, can't stand him.
Perhaps he realized that he was wrong and that's why he didn't eject your goalie when he took a player down on a break away.