Ok, I found out what the problem was. There wasn't one.

It seems that the factorio server has NAT loopback-problems:
When trying to connect to the public IP, while at the same time being behind that public IP, it doesn't work and behaves as described.
But when trying to connect from somewhere else ...

I have successfully installed a (headless) factorio server in my home network. I can connect to it via "Play on LAN" (the server is found) and I can also use "Connect to server" by entering the private IP and port.
This works fine.

Now to the part that does not work: Connecting via a public IP ...