[0.12][cube] Headless server not working with more than one

[montoyo]

Related posts:

• https://forums.factorio.com/forum/vie ... =7&t=13558
• https://forums.factorio.com/forum/vie ... =7&t=13634

I thought it was because I was using two computers using the same router (= same IP) to join a remote Ubuntu server. The problem is that I tried with a Windows remote server and with a remote client (by remote I mean a machine with a different IP/which doesn't share the same router), and it doesn't work better The moment a second client joins the headless server it stops and says "waiting for player <unknown name>". This message closes only when this new client disconnects and displays an error message. I won't re-post the logs since you can find them in the latest messages of the first related post, and besides they do not contain helpfull things...
If it works for you guys please tell me if you're using IPv4 or IPv6 because I have reasons to believe that it's all related...
Thanks!

[daniel34]

I've set up a dedicated headless server to test 0.12 in a public setting.
Thread here: https://forums.factorio.com/forum/vie ... 53&t=13579

I connected there with no problems the first time and set up some smelting for iron and copper, then I went to sleep.
Today I can't join the server because I'm always getting the message Following peers are not responding: unknown username (see image on first page of thread).
I can only connect if I restart the server and then connect as the first peer, but it seems that other peers can't connect then.
Some people have reported that they can't connect to the server, others say that they had a fine time playing on the server with no problems.

A few minutes ago I changed my router settings so that port 34197 UDP + TCP are directly forwarded to my pc and also tested it and it was working, at least TCP (i ran an apache web server for a short time on that port and it worked fine when connecting from outside).
Still, I can't connect to my server and if I can because I'm the first to connect, it seems that others can't connect.

The client error log says:
Error MultiplayerManager.cpp:107: MultiplayerManager failed: "Failed to establish connection with peer 79.x.x.x:34197."
This is not the server ip and I've no idea why it would connect to a random peer on that port.

It seems it wants to connect to all other peers on that port, but I assume most won't have that port forwarded.
The log is also on the second page of the thread I linked.

EDIT: further system info
The server is a dedicated linux server running Debian 8.1 x64 with an Intel Xeon CPU W3530 @ 2.80GHz quadcore processor, 16 GB of RAM and a 1Gbps/250Mbit (down/up) connection.
I've had to install some dependencies (libsound and libx11 amongst them IIRC) first, but when I connected for the first time it was running without any problem and no noticeable lags.
Port 34197 is open for both UDP and TCP.

[montoyo]

So I ran some tests here are the results:

And as you can see these are confusing... I think it only happens when the server is headless and on a different network... odd

[Oxyd]

Error MultiplayerManager.cpp:107: MultiplayerManager failed: "Failed to establish connection with peer 79.x.x.x:34197."
This is not the server ip and I've no idea why it would connect to a random peer on that port.

It seems it wants to connect to all other peers on that port, but I assume most won't have that port forwarded.
The log is also on the second page of the thread I linked.

Because it's a peer-to-peer architecture, all peers have to be able to connect to every other peer. If someone doesn't have port 34197 forwarded multiplayer won't work for them.

Is there any firewall active on any of these computers? Is it possible it's dropping the UDP datagrams?

Also you're reporting this as a headless bug. Does this mean that all of this works when you run Factorio normally – that is, not headless? Your table shows only the last test to be non-headless.

[daniel34]

The table above is not from me, it's from montoyo.

When talking about a dedicated headless server, I thought that there is a centralised server that handles all the connections between the clients.
So that every client only connects to the server and not to other clients (star-topology).

From my understanding, a dedicated headless server is used so that peers behind NATs and firewalls can connect to a game and the server handles all the rest. But with factorio, the server itself isn't a server per se, it's just an additional client on the current list of peers. I've had it happen that my internet connection would drop during a multiplayer game and my client would say that the <server> was dropped from the game. The game continued like I was the host, but because I was not connected it was like singleplayer. i.e. I've forked the game, so that my client and the server have different gamestates without knowing about each other.

Like in most other games I expect that when I run a dedicated headless server that the server always works as host, and all peers only connect to that host, even if they are behind firewalls/NATs. And should you lose the connection to the host, you would be dropped from the game. There should be no interaction between the clients.

From your answer I infer that every peer joining a multiplayer game has to have port 34197 open (TCP/UDP?) so that the P2P architecture of factorio can work. In my opinion this defeats the purpose of a dedicated headless server, where everyone can connect as long as they have some kind of internet connection. Right now the server just works as an additional peer.

EDIT: Right now most of the factorio players are rather tech-savvy, but as soon as it will be released on steam, a lot of people will come and wonder why multiplayer doesn't work for them. (i.e. port not forwarded).
I think that currently it isn't possible to play multiplayer factorio unless your 34197 port is open and forwarded to the correct machine. Please correct me if I'm wrong about that, but I got that from your answers.

Is there any firewall active on any of these computers? Is it possible it's dropping the UDP datagrams?

I've started this public server because of the assumptions above (client only connects to server, doesn't need to have ports open), so I can't tell you about all the computers participating, but I'd guess some wouldn't have port 34197 open.

[Oxyd]

Yup, that's exactly how it works. (Also UDP – Factorio uses UDP for all of its multiplayer.)

Star topology might come in a future update.

[daniel34]

The problem is that I can't join my own server because of this error:


Like you said, every peer has to have their port 34197 UDP open so that they can play the multiplayer session.
Wouldn't you be able to check on connect whether the connecting client has their port open, so that if there's no connectivity for that client you drop that client?
I currently can't join the multiplayer session because I can't connect to 79.x.x.x:34197.

It would be much more user-friendly to just tell that client that they can't connect because their port isn't opened.
Right now this prevents me and some other people to play on that server, just because one/more peers are connected that haven't opened that port.

[Oxyd]

The way you check whether a port is open is by requesting something to be sent to that port and then you wait and see if it actually arrives. Which essentially is what Factorio does. Or should do, anyway.

These peers should drop out and be removed from the game when the progress bar reaches the end. I take it that doesn't happen, then? Also all of these peers are in the game with no issues?

[Oxyd]

Actually, sorry about that, I was confused there for a while. To clear this up: Your server is up and running, there are people connected to it and they can play just fine. But you can't join the game because you can't connect to some other players. Is that right? This means that when the progress bar reaches the end, your connection attempt fails and the game goes on without you, right?

[Boogalo]

Thats' right. With port forwarding, everyone in that port goes to the headless server. If you try to connect as well from another computer on the same network behind the port forward, you can talk to the server, but all traffic coming into your network on that port goes to the server, so the computer you're trying to play from will never get any traffic from the outside players. Example of a game that works well: Terraria, but its much less complicated on the netcode side i'm sure. I have a terraria-server exe running on another pc on my network, listening on ip:7777. I can connect to that ip:PORT from my gaming pc ok, and people outside can connect to it ok through the forwarded port on my router. I understand the problems we're having are just a limitation of the factorio current peer-to-peer netcode.

[Oxyd]

You can try editing config.ini on the computer you're trying to play on (not the server). Change the port value to something other than the default – for instance, set port=34198. You might have to set up port forwarding to forward 34198 to your computer.

And yes, this seems to be yet another limitation of our multiplayer architecture. :/

[daniel34]

I've found the problem. Well, at least my problem, maybe not OPs.
I've set up my modem/router so that all traffic on port 34197 UDP+TCP is going to my PC, and I thought it worked because TCP worked just fine when testing, I just didn't know how to test UDP (that's what factorio uses).
It seems my router doesn't handle UDP traffic particularily well, because factorio didn't work.

I've found out that my router has a DMZ setting where all traffic coming in is by default routed to one specified PC. Additionally, it actually assigns the external IP you get from your provider to that PC. After I enabled that, Factorio worked like a charm in multiplayer. I was able to connect to my server and had a fun time just marveling about what other players have created on it.

PLEASE NOTE: Should you have the same problems as me and your router has the same settings, be aware that not only does it forward traffic for Factorio, it forwards ALL the traffic coming in to your PC. Make sure you have a properly configured firewall and anti-virus on your PC.

The only thing that bugs me is that the error message is Following peers are not responding: unknown username, ...
Then the dialog pops up and says Failed to establish connection with peer x.x.x.x:x
It makes it look like the server is running but other peers are not responding, although the problem lies on your end.
I've suggested before that after connecting the server should check whether the client's port is open before proceeding any further.

EDIT: to clarify, the server is not on my network, in fact it is hosted in France, while I'm in Austria. So changing port doesn't apply here.

[Rein]

I would just like to mention that my friends and I have set up a dedicated headless server that is off-site and we've run into this same issue as well. None of us are on LAN with each other or the server and some are unable to do port forwarding, and even some of the ones who *do* have their ports forwarded correctly can't join. It's a miracle if we can get even two people in the game together.

I know multiplayer was an afterthought but peer-2-peer is not a viable solution. When this game launches on Steam people are immediately going to try to play with each other and the immense difficulty (and luck!) involved in doing so will be a major problem. I know that re-doing the netcode will be a major effort. Personally I think it would be worth it for you guys to do so, but I'm not privy to the inner-workings of your team so I can only suggest that you cease all non-networked development for the time being and just focus on getting the game's multiplayer functioning up to the standards people on Steam are used to.

The alternative is to remove multiplayer functionality entirely. I really like this game and I want to see it succeed, but a broken multiplayer experience will turn a lot of people away.

Best of luck.

[cube]

Please can you verify if this is caused by at least one peer being connected through an internal IP (typically 10.0.something or 192.168.something) and one through the internet?
I attempted to write a little explanation how things (don't) work in this case: https://forums.factorio.com/forum/vie ... =10#p93014

[Rein]

No peers are connected through an internal IP. The server is located in a data center in California while the three players (including myself) we tested the most thoroughly with were in Texas, Virginia, and Pennsylvania. None of us were using a VPN of any sort.

The Texas player and the Virginia player both had their ports forwarded correctly. The Pennsylvania player did not have the ability to forward ports.

We tested as such:

Pennsylvania joins first -> Success
Pennsylvania joins first, Virginia joins second -> Success (but failed in 1/3 attempts. Nothing was done differently)
Pennsylvania joins first, Virginia joins second, Texas joins third -> Failure
Pennsylvania joins first, Texas joins second -> Failure

Texas joins first -> Success
Texas joins first, Pennsylvania joins second -> Failure (but did succeed in one attempt. Nothing was done differently)
Texas joins first, Virginia joins second -> Failure

Virginia joins first -> Success
Virginia joins first, Pennsylvania joins second -> Success (also failed in 1/3 attempts. Nothing was done differently)
Virginia joins first, Pennsylvania joins second, Texas joins third -> Failure
Virginia joins first, Texas joins second -> Failure

Each one of these steps was attempted several times each.

It seems pretty obvious that Texas is the problem child here, but that player double and triple-checked that his ports were forwarded and he had done everything short of setting his desktop as the DMZ (which would have been an exceptionally bad idea). Also he is a Tier 2 IT Support, so I trust that he knows what he's doing.

EDIT: We had a couple other players on from various locations/countries and with varying degrees of success for each, but for the duration of the testing described above it was only the three players (plus the headless server of course).

[cube]

quote

[quote="Rein"]No peers are connected through an internal IP. The server is located in a data center in California while the three players (including myself) we tested the most thoroughly with were in Texas, Virginia, and Pennsylvania. None of us were using a VPN of any sort.

The Texas player and the Virginia player both had their ports forwarded correctly. The Pennsylvania player did not have the ability to forward ports.

We tested as such:

Pennsylvania joins first -> Success
Pennsylvania joins first, Virginia joins second -> Success (but failed in 1/3 attempts. Nothing was done differently)
Pennsylvania joins first, Virginia joins second, Texas joins third -> Failure
Pennsylvania joins first, Texas joins second -> Failure

Texas joins first -> Success
Texas joins first, Pennsylvania joins second -> Failure (but did succeed in one attempt. Nothing was done differently)
Texas joins first, Virginia joins second -> Failure

Virginia joins first -> Success
Virginia joins first, Pennsylvania joins second -> Success (also failed in 1/3 attempts. Nothing was done differently)
Virginia joins first, Pennsylvania joins second, Texas joins third -> Failure
Virginia joins first, Texas joins second -> Failure

Each one of these steps was attempted several times each.

It seems pretty obvious that Texas is the problem child here, but that player double and triple-checked that his ports were forwarded and he had done everything short of setting his desktop as the DMZ (which would have been an exceptionally bad idea). Also he is a Tier 2 IT Support, so I trust that he knows what he's doing.

EDIT: We had a couple other players on from various locations/countries and with varying degrees of success for each, but for the duration of the testing described above it was only the three players (plus the headless server of course).[/quote]

Could you please post server and texas log form texas joining second?

[creodor]

I'm 'Texas' from Rein's comment, and also the owner/operator of the server in question. The server is physically located in Las Vegas, Nevada; it's out of state for everyone involved, no local networks involved at all.

As requested, I pulled the logs from both the server and my client. Neither was set for verbose logging, and I'm not 100% sure anymore that these are matching logs. We can and will perform the tests again and provide better match-sets of logs for you to look over, but we won't be able to do that until later tonight when everyone involved is available.

Full logs are attached to this post; the client log was too big for pastebin and similar sites.

I noticed a specific segment that seems to be the source of the problem, and it doesn't *look* like it's port forwarding related at all, but something else going wrong. I could be incorrect, of course, but as said before I've verified multiple times that port forwarding is correct. And in advance, no, I will not set my computer to DMZ to verify this. Anyways. What I noticed (this is on my client side) is that after the map is downloaded, state changes are completed, and it's then trying to add peer 2 (Pennsylvania) to my client's game, the client mixes up peer 2 with the server:

124.972 Info MultiplayerManager.cpp:1160: networkTick(3218) mapTick(1260413) stopping mapAlign
124.972 Info MultiplayerManager.cpp:1654: performing map align task (SendPlayerJoinGameAlignTask)
124.972 Info MultiplayerManager.cpp:848: networkTick(3218) mapTick(1260413) changing state from(InGameWaitingForOthers) to(InGame)
124.972 Info MultiplayerManager.cpp:1470: networkTick(3218) mapTick(1260413) peerID(1) fullStateLog: local state(InGame) local peers(((peerID(0) state(InGameWaitingForOthers) mapAlignTick(-1))
((peerID(1) state(InGameWaitingForOthers) mapAlignTick(-1))
)
125.044 Info MultiplayerManager.cpp:989: networkTick(3224) mapTick(1260418) received stateChanged peerID(0) oldState(InGameWaitingForOthers) newState(InGame)
125.044 Info MultiplayerManager.cpp:989: networkTick(3224) mapTick(1260418) received stateChanged peerID(1) oldState(InGameWaitingForOthers) newState(InGame)
125.099 Info NetworkInputHandler.cpp:703: mapTick(1260422) networkTick(3228) connecting to player(creodor).
125.099 Info NetworkInputHandler.cpp:561: assigning playerIndex(0) to peer(1)
125.099 Info GameActionHandler.cpp:1913: MapTick(1260422) processed PlayerJoinGame peerID(1) playerIndex(0) mode(connect)
175.444 Info Router.cpp:648: networkTick(6235) adding peer(2) address(69.249.xxx.xxx.34197) sending connectionAccept(false)
175.444 Info Synchronizer.cpp:491: networkTick(6235) adding peer(2) success(true).
175.535 Warning Router.cpp:202: Parsed peerID(2) expected address(69.249.xxx.xxx.34197) doesn't match received address(23.226.xxx.xxx:34197)
175.564 Warning Router.cpp:202: Parsed peerID(2) expected address(69.249.xxx.xxx:34197) doesn't match received address(23.226.xxx.xxx:34197)
175.570 Warning Router.cpp:202: Parsed peerID(2) expected address(69.249.xxx.xxx:34197) doesn't match received address(23.226.xxx.xxx:34197)

Verified with the user that 69.249.xxx.xxx is indeed our Pennsylvania player, and I verified myself that 23.226.xxx.xxx is my server. From what I see in the log, and my admittedly basic understanding of how the networking is handled, it looks like my client is mixing up the server and our Pennsylvania player. I have no idea why that might happen.

Initially I thought maybe it was a CRC error with the map download, but during the dropout of peers, the game confirms that CRC matches the expected value.

After dropping peer 2, it does the same thing for peer 1 (Virginia), saying it's getting the server's IP instead of the expect value.

Just to forestall statements of 'oh you just don't have port forwarding setup right', I took screenshots:

(For clarity, 34197 | 34198 is the port range, start | end. I cut that part off in the screenshot. I have more than one server running, to test things, but these logs and tests were done on the main non-testing server instance)

[Bisa]

(For clarity, 34197 | 34198 is the port range, start | end. I cut that part off in the screenshot. I have more than one server running, to test things, but these logs and tests were done on the main non-testing server instance)

just out of curiosity, which port is your client using? As you mentioned you have other servers running for test purposes - does shutting them down and ensuring your client is the only active game on your local network have any effect ont he outcome at all? (just throwing it out there, not really expecting it to have an impact but who knows )

[creodor]

I just have both ports forwarded for occasions where I run and connect to the test server instance. While diagnosing this, only client/server instances using port 34197 were running; 34198 was unused during all of this, I was merely clarifying why that port was in the open range.

[Rein]

Cube: We have redone our testing and saved logs for each test. Note that the Server was restarted after each test. The label format is as such: [Test #] - [Log source] - [Order of joining map 1st-2nd-3rd]

These are the users, their locations, IPs, and the number of times their IP showed up in all 20 files:

Creodor (Texas): 64.92.xxx.xxx - 33 hits 14 files
Rein (Virginia): 100.36.xxx.xxx - 2433 hits in 10 files
Dakkan (Pennsylvania): 69.249.xxx.xxx - 2434 hits in 9 files
Server (Nevada): 23.226.xxx.xxx - 4844 hits in 14 files

Below are the logs for our tests as well as the number of times each IP shows up in each file:

Test 1 - Creodor - Dakkan-Rein-Creodor http://pastebin.com/UdL7xuxF (Server:1210, Dakkan:605, Rein:605, Creodor:0)
Test 1 - Dakkan - Dakkan-Rein-Creodor http://pastebin.com/WmN133NS (Server:2, Dakkan:0, Rein:1, Creodor:1)
Test 1 - Rein - Dakkan-Rein-Creodor http://pastebin.com/ZqJCrjEw (Server:1, Dakkan:0, Rein:0, Creodor:1)
Test 1 - Server - Dakkan-Rein-Creodor http://pastebin.com/TnREWC8T (Server:0, Dakkan:4, Rein:4, Creodor:4)

Test 2 - Creodor - Dakkan-Creodor http://pastebin.com/ZVZWWqrb (Server:607, Dakkan:607, Creodor:0)
Test 2 - Dakkan - Dakkan-Creodor http://pastebin.com/W9ULyPGk (Server:3, Dakkan:0, Creodor:2)
Test 2 - Server - Dakkan-Creodor http://pastebin.com/1yciRvqW (Server:0, Dakkan:4, Creodor:4)

Test 3 - Creodor - Creodor-Dakkan http://pastebin.com/BseZa0C9 (Server:601, Dakkan:601, Creodor:0)
Test 3 - Dakkan - Creodor-Dakkan http://pastebin.com/m2Tr8Fz9 (Server:1, Dakkan:0, Creodor:1)
Test 3 - Server - Creodor-Dakkan http://pastebin.com/h2MhSdeK (Server:0, Dakkan:4, Creodor:4)

Test 4 - Creodor - Creodor-Rein http://pastebin.com/M1aF2DaR (Server:601, Rein:601, Creodor:0)
Test 4 - Rein - Creodor-Rein http://pastebin.com/cgSPbqxP (Server:1, Rein:0, Creodor:1)
Test 4 - Server - Creodor-Rein http://pastebin.com/FtpaP2bV (Server:0, Rein:4, Creodor:4)

Test 5 - Creodor - Rein-Dakkan-Creodor http://pastebin.com/tQB0zqJY (Server:1209, Dakkan:604, Rein:605, Creodor:0)
Test 5 - Dakkan - Rein-Dakkan-Creodor http://pastebin.com/WsjHbFTN (Server:2, Dakkan:0, Rein:0, Creodor:1)
Test 5 - Rein - Rein-Dakkan-Creodor http://pastebin.com/1AV8Gnz7 (Server:1, Dakkan:1, Rein:0, Creodor:1)
Test 5 - Server - Rein-Dakkan-Creodor http://pastebin.com/CLGimLXU (Server:0, Dakkan:4, Rein:4, Creodor:4)

Test 6 - Creodor - Rein-Creodor http://pastebin.com/f8ZDFrCA (Server:604, Rein:604, Creodor:0)
Test 6 - Rein - Rein-Creodor http://pastebin.com/ys5A37m1 (Server:1, Rein:0, Creodor:1)
Test 6 - Server - Rein-Creodor http://pastebin.com/mT6UdtnC (Server:0, Rein:4, Creodor:4)

Also, Creodor has supplied screenshots of his IP forwarding setup, which are attached.