Along the same lines as your question about injuries, how about this one: do you feel that a referee is responsible for ANY injury on the field? Even those where absolutely no foul has occurred?
If referees decide they will be responsible for every injury that might occur on the field, I believe they will start second-guessing everything they do. If that happens, I believe many matches will never end, because the whistle will be constantly blowing.
Now, if you do make sweeping changes as to how/when referees make "dangerous play" calls, i.e., tell referees to make the calls based mostly on goalkeeper safety, I believe you will then see many/most goalkeepers take advantage of that - they will start putting themselves into more "dangerous" situations, knowing the referee is going to make calls in their favor.
That could potentially lead to a worse situation than now exists. Just a thought...
