I don't really think money is the solution.
In a day where a ref works a JV & V set of matches, I believe the pay is probably in the range of $70-$80 for the set, plus mileage. Remove the mileage and deal with only the actual payment for the games, even on the low end $70 for the night.
JV game, 1 hour and 10 min of "work"
Varsity game, 1 hour and 30 min of "work"
pre and post game, 1 hour of "work"
Works out to be approximately $19 per hour. What other "part time profession" requires 1 hour of classroom instruction, pass a test, and less than $50 of admin costs each year, but pays more than $19 per hour?
Many who hang it up tire with the assignment process, working a terrible 2 man system, and spectators who feel it's their right to demean the referees.
Fix the people and the social aspect of how they attend the games, and what is deemed to be acceptable behavior, and you may fix the issue. Each year, in addition to soccer matches, I attend several HS football games and basketball games, and with the exception of the rare instance of the "loud mouth" person demanding the refs head, it's not the same. But attend any soccer match, and often times it's all you hear. Better yet, go to what you feel is a competitive soccer match in your area, where you don't really have a personal connection, and just watch and evaluate how you are as a spectator, compared to those that night. It can be eye opening.