[1.1.25] Extremely lag and game freeze/pause after 1.1.25 update

[kaavik]

Hello,

We love your game! Which is why I am taking the time to write this bug report because I really want to play but with this new bug from 1.1.25, it is LITERALLY unplayable - my friend gets dropped from multiplayer because of this new bug/problem!

###

My friend and I have been playing Factorio multiplayer on the same map for over a year. We know how our map performs and I always host (he usually gets 60ups/fps while I usually get 30-35ups/fps). After the update to 1.1.25, when the map first loads for me, I notice that I am getting normal FPS of ~35 and then as I ride in my train to the destination (game was saved while riding somewhere), the FPS slowly lowers and then once getting to my destination (NOTE: map save name is Malone 135a) the FPS is ~24 and it starts to freeze, dropping to single digits at times. It spikes back and forth, occasionally back up to low 30's or high 29's but then mostly stays lower and keeps on pausing/freezing - even when FPS/UPS would be showing 30+ FPS.

I play on Mac. My friend plays on Windows. We both are experiencing the problem including freezing / pausing when playing the game locally - with no one else joined. We also play 3 three mods (bottleneck, chunkychunks, and infinizoom) but we both disabled all of the mods and still getting the same behavior.

For me, the FPS degradation and freezing/pausing also seems to be consistently reproducible when loading from the save game Malone 135a and doing nothing except riding the train - so there is no variance in actions taken once within the map and all of the same calculations and events happen the same each time (unless you have randomness involved in your code for enemy spawn, etc).

I am attaching a screenshot showing ~12UPS/FPS even though it dips lower at times and even when ~30+ still does freeze/pause events (super annoying!). The problem became bad enough tonight that my friend (Malone) was getting dropped from the multiplayer game and eventually did actually get dropped - and there was nothing either of us could do about it.

One other thing that might be a clue - normally, when he joins, I pause the game to allow his "catch up" phase to be extremely quick. Usually, when I click pause while he is joining, my FPS/UPS jumps from ~35-ish to 60/60 while it is on pause. In the recent case since updating to 1.1.25, I noticed that the FPS/UPS no longer match and it is reporting an FPS of ~50 or even less while UPS stays at 60. I don't recall that ever being the case in the past so maybe that will also help.

If I were to describe how the problem FEELS, it feels like there is an issue/slowness that is building until it reaches a tipping point and then "resets" and then starts over with FPS being "normal" and then gradually drifting down to some problem threshold where the game freezes (or in Pause mode just gets down to ~50fps) and then pops back up to the normal (~35 FPS unpaused and 60 FPS paused) value and starts over.

I am including screenshots showing UNpaused gameplay with low ~12 FPS/UPS and second screenshot of PAUSED gameplay showing 48/60 FPS/UPS.

Again, we love your game and hope you can quickly find and fix this bug!

kaavik

[Loewchen]

Post the save please.

[kaavik]

I have tried multiple times to attache the save game (Malone Day 135a.zip 116.2 MB) and it keeps failing. Here is a link to the file in my Dropbox:

https://www.dropbox.com/s/2xbo9j60h7nit ... a.zip?dl=0

My username is kaavik and when I load this game, I am riding in a train back to the base mall area. By the time the train gets back to the train stop, the FPS will have dropped from ~35 FPS at launch of the save to 24 FPS when I get to the base and the game starts freezing. If you just sit in the train and do nothing (just uses map view to scroll around the map, or use the game without even trying to move / walk around - or walk around - it doesn't really matter, the game FPS/UPS will fluctuate all over the place and even when it might be showing ~30FPS, all of a sudden you'll get a 0.5 - 1.5 second freeze where the whole game simply freezes briefly. It is very annoying because it doesn't seem predictable and in the past when we had +200K logistics robots actively flying around, my FPS would be ~18 but it was always consistently slow (18FPS) and there still wasn't any of the freezing that we are seeing now with all of the FPS fluctuations and random freezing.

My friend, Malone, agrees that when he hosts the save directly on his very nice gaming PC, the performance degrades until it hits some invisible threshold and then "resets".

I play on a maxed out 2020 Macbook Pro 16" with i9, 32GB RAM, 2TB SSD, with the upgraded AMD Radeon 5500M video card. Again, this has never been a problem prior to 1.1.25. Also, when I check the CPU/Memory/GPU stats, the CPU is low single digits, memory is less than 50% utilization, and the GPU doesn't seem too taxed either. The problem happens whether I am playing alone or have my friend connected in multiplayer.

[Rseding91]

When I load the save and run it; virtually all of the time is spent in train re-pathing as train stops are enabled/disabled. Looking at the save; that isn't too surprising given how much rail and train stuff you have made.

Do you have a version of the save that runs fine in an older version of the game but lags in 1.1.25? I suspect you simply turned on all the trains and now you're seeing the full impact of what you've built. But if you have such a save I can take a look at it.

[kaavik]

FYI, as a long-standing user that did all of the things that were asked to submit a bug report and even provided what I would consider a lot of detail and particularly mentioned that we have been playing for a long time (more than a year now if I'm not mistaken) on the same map, to make the comment "I suspect you simply turned on all the trains" feels like a condescending response to a very frustrating problem. And, as you point out, we have a lot of trains in a large base which certainly didn't happen overnight. Extremely disheartening response as a Factorio fan.

Regardless, I am attaching a save game from 1.1.24 which did not exhibit any of the problems and still had nearly as many trains. Based on your response, maybe there is a threshold such as 200 train limit and we were at 197 up until 1.1.25 and now we have 202 and the game can no longer keep up with the pathing. I can believe that is a possibility but I have also seen reddit posts of maps with 800 trains and so I would not have assumed ~200 trains as a magic number for the game to start crashing. I guess based on your feedback, I can delete a few trains and see if the performance recovers.

Link to Malone Day 133 save game which runs on 1.1.24 and did not exhibit the freeze problems and probably had just under 200 trains active and "turned on":

https://www.dropbox.com/s/0qtbonluq2m5c ... 3.zip?dl=0

NOTE: We did try to go back to 1.1.24 ourselves but Steam did not allow us to go back to 1.1.24 and still use our most recent save from 1.1.25 (we even lost all of our achievements.)

[Rseding91]

I understand your frustration but I can see the total code difference between 1.1.24 and 1.1.25 on a single screen and none of it touches trains in any way.

1.1.25 had exactly 2 game logic (entity) changes:

1. A crash related to canceling invalid upgrades when removing mods (changed construction robot and entity with health in those migration cases)

2. A bug with cross-surface electric pole connections adding extra connections when removing specific electric poles. (only triggers when using cross-surface wires and doesn't have any crashes/performance implications - just not-desired behavior)

So what ever is happening isn't related to any code changes.

[Rseding91]

The key difference I'm seeing between these saves is: the 46 trains you have moving iron ore around are almost all no-pathing in the 135a save. Every time the stop they want to go to turns on/off all of them try to re-path and that hammers the game slowing it down.

When I re-enable any one of the iron-mine rail stops all of the trains resume moving and there's no more stuttering.

[kaavik]

I can appreciate that the issue may not actually be specific to 1.1.25 and it could be pure coincidence that we happened to hit a performance threshold which became apparent at the same time as our update to 1.1.25. I will accept that 1.1.25 is not the problem.

That said, it does not change the fact that there *IS* a problem which drastically affects gameplay to the point that remote users via multiplayer are getting dropped. I imagine this would not be desired - and hopefully not even acceptable - to the Factorio developers so my bug report remains:

Extreme lag and game freeze/pause causing multiplayer drops and very poor user experience in single player mode.

I request that the developers would please still into this issue, maybe in a larger context of some threshold / pathing limitation / bottleneck / code "whatever" that is causing problems for multiple users (me and my friend) on multiple platforms (mac and PC) whether playing single player or multiplayer.

As I've already said, I love Factorio and have put a lot of hours into this game - more than any other game in my 45 years of life - and would really like to see the performance issue addressed in such a way that it does not cause the game to freeze and/or drop players during multiplayer.

[kaavik]

The key difference I'm seeing between these saves is: the 46 trains you have moving iron ore around are almost all no-pathing in the 135a save. Every time the stop they want to go to turns on/off all of them try to re-path and that hammers the game slowing it down.

When I re-enable any one of the iron-mine rail stops all of the trains resume moving and there's no more stuttering.

This is GREAT information... so what would you recommend to keep it from happening while maintaining some logistics logic to prevent all trains from heading to a single stop with no limit?

NOTE: As things are, I make use of a "railyard" to try and get all of the trains to a "safe" place for them to "No Path" without blocking main lines - but it doesn't seem like the railyard would help with the spike in re-path calculations when the train stop status changes from no path for many trains to *A PATH* for many trains.

Would love any suggestions as I am happy to try and adapt/accommodate from a design perspective as well.

[kaavik]

@rseding91 - Also, is there an easy way for us regular players to quickly identify which trains/stops might be causing this No Path / recalculation problem with large numbers of trains and stops?

[NotRexButCaesar]

[...]

I know this isn’t your point, but:
Have you thought about setting the train limit to 0 instead of disabling the station? It should prevent the re-pathing and improve performance.

[Rseding91]

[...]

I know this isn’t your point, but:
Have you thought about setting the train limit to 0 instead of disabling the station? It should prevent the re-pathing and improve performance.

This is the solution to the performance impact. Instead of turning the stop off set the limit and it should work as you want; when the limit goes up; only 1 train will try to path to the stop instead of *all* of them.

As for the performance issue; I'll look into it but I don't know if there's anything that can be done. Train path finding is quite expensive at the scale you've built.

As for how to find which ones; it's about how many trains are doing the path-finding. I just looked at the largest group of trains in the trains GUI and saw most of them were status: no-path and that the no-path ones said all the stops were disabled.

[kaavik]

I know this isn’t your point, but:
Have you thought about setting the train limit to 0 instead of disabling the station? It should prevent the re-pathing and improve performance.

I've already used train limit 0 at times but your suggestion is unclear to me because I believe if the train limit is 0, won't it still count as a No Path (can't go to an active Stop that is already at limit (0 = no trains) so then it would still be a No Path situation until a Stop re-enables or becomes available due to less-than-limit trains there?

If I understand correctly, it won't actually change the root problem - a bunch of trains sitting around with no place to go and then suddenly having someplace to go when the mine has another load ready of the smelting cell is ready for another load of resources...

[kaavik]

I know this isn’t your point, but:
Have you thought about setting the train limit to 0 instead of disabling the station? It should prevent the re-pathing and improve performance.

This is the solution to the performance impact. Instead of turning the stop off set the limit and it should work as you want; when the limit goes up; only 1 train will try to path to the stop instead of *all* of them.

@Rseding91 - I am surprised by this answer, meaning, it seems like I must not fully appreciate the difference between an disabled train stop and an enabled train stop that is full (limit met either because set to 0 or because set to 3 and already have 3 trains on the way).

Based on the answers from you and @AmericanPatriot, I would have expected the pathing logic/behavior to be the same for both, meaning that when a Stop becomes enabled, all of the No Path trains immediately try to path to it and if a Stop was full and now has capacity for another train, that all of the waiting trains with No Path or Destination Full would immediately try to repath.

Please clarify for me. Thanks in advance!

[Rseding91]

Enabling and disabling train stops has a much larger overhead vs limiting and unlimiting.

Enabling a stop will cause every single train to try to path if they have that stop in their list and they aren't currently pathing; up until the stop limit is reached.

Disabling a stop will cause every single train in the no-path state to try to re-path because if you disable a stop for a train it may need to switch to the next stop in the schedule. So by turning off the last stop for a series of trains it forces every single one of them to try to re-path - regardless of what limit may be on any existing stop.

In this way; disabling a stop is incredibly expensive because it acts like the stop no longer exists and forces all trains to try to act that way. Just turning the limit down on a stop does nothing to existing trains without paths or trains currently pathing and so has no real cost.

At least; as far as I understand how all of this is currently working. I'm going to ask one of the train guys to look at this because it seems really weird how it's working now.

[boskid]

I looked into that save file and it is "not a bug".

96417.png (1.28 MiB) Viewed 4699 times

There is a missing curved rail around {1171, -917} that is leading to a `Iron Mine` train stop. Because of that there are literally 42 trains that are no-pathing. I cannot prevent those trains from repathing because missing rail could be only isolating some trains while others could still find a path. When a pathfinder makes a no-path, it is one of the worst performance cases because the pathfinder has to touch literally every reachable rail segment only to say "i was unable to find a path using that set of rail connections" so there are no surprises for me that it was slow as hell. After adding missing rail piece there were no performance issues in that save file.

[kaavik]

Boskid,

Thank you for the additional information and detail about the specific missing piece. I guess it also makes a lot of sense that you would have to look at each track segment and with the size of our base, number of trains, stops, and track segments, could create a decent sized burden for your pathing calculations.

I will definitely install the missing track segment next time I get to play and am looking forward to not having the game freeze any more!

Thanks for your additional help, Boskid!

NOTE: It sounds like you are THE train developer and I just want to say that the trains are definitely my very favorite part of this game! Super well done!!!

[kaavik]

Enabling and disabling train stops has a much larger overhead vs limiting and unlimiting.

Thank you for explaining this to me! Extremely helpful, Rseding91!

[kaavik]

I looked into that save file and it is "not a bug".

I just wanted to confirm that the combination of building the missing track piece and reducing the number of trains for that series of stops (46 -> 27), did fix the problem with the game randomly freezing. Thanks for your help troubleshooting and explaining why the train logic in Factorio was causing the problem!

Keep up the great work!

kaavik