When referees decide who they will/will not officiate, there's a problem. That should be purely up to the assigner.