I wouldn't say that you have to be an idiot to ref. Though, I will agree that in a sense, refs can't win. If you think about it, they can only make half the people happy, half of the time. It's really a lose-lose situation for them.
That said, I would not say they were idiots. (Given, walking into a lose-lose situation 'just because' would be a bit idiotic.)
In my opinion, ref's do not go out onto the field 'just because'. It's the love for the game itself. If they didn't truly love soccer (or just officiating), why else would they stay out there and take all the hateful remarks said to them? And even after they hear hateful or downright rude comments sent in their direction, they still hold their head high. It takes alot of good attributes to make a good ref but I don't think idiocy is one of them.
There are alot of very kind, down-to-earth refs out there. It's a shame that we are faced with a shortage of these kind of people to ref our high school games. Though, if you think about it and look at all the abuse they are put through, can you blame them for avoiding the situation, thus, avoiding being a ref altogether?