My friend has port forwarded it correctly. We believe this is something on my end.

So I'm trying to connect to my friend server, and the other 4 people can get on fine. But when I join, first I don't download a map, and second when I join when other people are on, it ends up making them disconnect and reconnect, I don't even get in. When its just me and the server owner on, I can ...