That's my point. It's easier to officiate a higher-level match when the kids are playing the ball and you have quality players with experienced (well-behaved) fans on the sidelines.
When you have a high school varsity boys "backyard brawl" (rivalry game), and the match gets physical, and perhaps chippy, the referee doesn't look so good. Testosterone flares! Passion is high and everyone's vision (perspective) gets skewed.
I've had coaches yell at me, "Come on ref, swallow your whistle and let the game flow". Sixty seconds later the same coach yells, "Geezus ref, I can't believe you missed that. Call something!".