Oz and all,
Did you suspect that SV players were goin to start fights with you? No, It seems that you didn't trust yourselves.
By not going through the ritual of sportsmanship that is the after game hand shake, you dishonored the game, your opponents (which I think is the real reason you wouldn't go out) and, most of all, you dishonored yourselves.