I've seen referees wear black, yellow, red, pink, etc. Doesn't bother me a bit. As long as you can differentiate them from the players, what's the big deal?

What I don't like is when teams don't have offsetting colored socks. I thought that was a rule? If teams have to have different colored jerseys, then they should have to have different colored shorts and socks!