I'm not positive, but you may have only black balled them from your home matches. There's a chance they could be assigned your match if on the road.
Something my AD mentioned to a coach from another sport when they asked about black balling, "If you black ball this one, will his replacement be better?" Definitely something to think about when taking that action. You would hope you've seen the worst official ever if you take that action, but there's always a possibility there's one worse out there you haven't seen yet!