I have known players (my own children) to be encouraged, by coaches, to play rough/physical till the whistle. Once the ref shows the players what they can and can't do they pull back. Problem is some players do not how to play physical without playing dirty.
Anyone who thinks refs do not impact the play is dreaming. The job of a ref is to impact the play, they are there to administer the game and make judgments. Failure to make call is much an impact as making them.

Good refs find a balance where they establish to the players how close they will call it while still letting the players play the game.