Best thing to do is to simply go to their Steam profile and block them. They will be unable to join any server as long as you are there. So far I have blocked 138 griefers, and it definetly has made a pretty big difference as I almost never come across a griefer these days.