While this happens to me on occasion, I believe it is something on the server's side that prevents connections from overly laggy sources, bad ping, inconsistent packet connections, etc.
More often than not, a server will lag for all players ever so slightly when new players join/when someone saves, and
stays a little laggy during the entire time a player connects. This is not an ideal scenario.
Server-savvy game starters will instead create blanket filters stopping anyone with
undesireable connections from joining. These include IP address filters, username filters, connection speed filters, etc. Some will allow you to join, and shunt you off depending on whether a more desirable connection joins.
As far as helping OP, I can only really reccommend a faster/more stable internet connection, faster computer. My similar problems I've had wih desyncs/freezing/crashes/lost connections vanished after replacing my aging 7 year old Samsung RC730.
Try changing your computer's power plan in Windows Mobility Center to Highest Performance.
I'd assign this to Not A Bug, but thats just me. Could actually
be a genuine problem I'm missing.