[0.15.x] Severe lag and lag spike issues

[xBlizzDevious]

Hello,

My friend and I are playing on a heavily modded server that I run. It's becoming unplayable for an unknown reason. I don't know how to do proper debugging, but I have enabled "show-time-usage" and investigated that screen.

No particular mod is obviously causing the issue. Instead, what I'm seeing is that:

Code: Select all

Frame cycle
>Wait For Update

Is what's spiking. See attached image for the time usage statistics.

Stats

HighUpdateWait.png (11.07 MiB) Viewed 11634 times

Wait For Update varies but is usually <10ms. When doing certain things, it starts rising and can easily reach 200ms or more. Said things include picking up items from belts or even just placing down buildings or belts in quick succession. Just replacing a single line of old yellow belts with reds is causing this issue.

Even worse, it has now become almost unplayable as it's no longer JUST spiking, it's ALSO becoming much more laggy when players are doing nothing. I would expect minor slow-downs when building a base that's larger than this, but in previous major versions of Factorio, I've had substantially larger bases that ran far smoother.

Please ask if you require any more details. World saves and a copy of the mods used can be sent if required.

[xBlizzDevious]

I would like to add to this that if I save the game from the server, then open it up in single player it runs flawlessly.

Even when I ask bots to remove a couple of hundred belts or I hold F and run along a belt.

After some experimenting, it appears that it may be a hardware fault with my server. Investigation underway. Hopefully this is the cause of it because I have no idea what else it could be.

[xBlizzDevious]

After a couple of days diagnosing, I have indeed got some faulty RAM. I've tested slots and modules - all slots are good and two of four modules are good. Two faulty modules have been removed.

However...

I've been playing Factorio on my main computer as a multiplayer game but mostly was on my own. The game works fine for just me though does occasionally slow to ~50 FPS/UPS.

I just started the server back up, copied across the current save and mods and started up the Factorio server.

All loaded OK so I hopped on my main PC and tried to join the server - catching up failed. The server is using slightly faster single-threaded hardware than my main PC so I assumed it was just running too fast. Tried a few times and eventually (it took about 20 minutes of catching up) it managed to get in.

I was immediately greeted with 5 FPS and 60 UPS. What? Quick check of "time-usage" shows that "wait-for-update" is still the culprit.

On top of this, when my friend tried joining while I was hosting on my main computer, it would often become unplayable for him.

Therefore I can only assume there is a bug somewhere with multiplayer (shock horror!). Quite possibly caused by mods but I can't test that as the game works fine until you start building big.
Now the base is reasonably big, but it's not huge so obviously there is something causing the issue.


These are all of the details I can think would be useful.

Server specs:
Intel Core i7 4790K 4GHz - 4.4GHz boost
16 GB Corsair XMS3 1600MHz Cl9 (normally 32GB but half is faulty).
Intel iGPU but running headless so irrelevant.

My PC specs:
Intel Core i7 3930K OCed but tested stable @ 4.4GHz.
32 GB Corsair Dominator 1600MHz Cl10 (I think)
Nvidia GeForce GTX 980Ti

Factorio 0.15.9 is current version I've been experiencing this on but it's been ongoing since 0.15.3 - simply because that's when the base started getting bigger.

I can provide a copy of the save and mods (some are tweaked from available versions and some are unavailable elsewhere as I made them (or ripped them from others)). Please request and I will PM details as they are hosted privately.



If anyone can advise on how I can track down the issue more myself, that would also be great.

Thanks!

[Caennanu]

Hey Mate,

I had similar issues in previous versions.
What solved it for me is downclocking the server to under 3ghz. my sweetspot seemed to be around 2.6ghz, but that is with an AMD cpu at 4ghz for game pc.
i'm expecting you might have a sweet spot a little higher.

Try downclocking the server to 3ghz, see what happens? (also try to disable the boost and set affinity for the server to 1 core)

p.s. if you use vmware or other virtualization software it may even be possible for you to assign say . . 60% of a core to the VM.

[xBlizzDevious]

Hey Mate,

I had similar issues in previous versions.
What solved it for me is downclocking the server to under 3ghz. my sweetspot seemed to be around 2.6ghz, but that is with an AMD cpu at 4ghz for game pc.
i'm expecting you might have a sweet spot a little higher.

Try downclocking the server to 3ghz, see what happens? (also try to disable the boost and set affinity for the server to 1 core)

p.s. if you use vmware or other virtualization software it may even be possible for you to assign say . . 60% of a core to the VM.

Thanks for the input.
It's a computer that I run a bunch of things on including virtual machines but I don't want to add more and put Factorio on one. That alone would probably cause it to run slower. It's base OS is Windows 7 and I run VMs from within that, I should note.

I'd also note that I shouldn't have to specifically downclock a computer to be able to play a game. The thought is just ridiculous!

And I don't think that would solve the horrendous framerate issues on the client which is more of an issue than not catching up.

[Caennanu]

I'd also note that I shouldn't have to specifically downclock a computer to be able to play a game. The thought is just ridiculous!

I completely agree, i thought the same, but it made the game playable for me.
The other alternative is to turn down game speed to say 0.8? just to see what happens.?

[xBlizzDevious]

I'd also note that I shouldn't have to specifically downclock a computer to be able to play a game. The thought is just ridiculous!

I completely agree, i thought the same, but it made the game playable for me.
The other alternative is to turn down game speed to say 0.8? just to see what happens.?

When running on a server, that seems to always cause issues for clients. Not sure what the problem is but it's probably that not everything runs at the same speed and it breaks.

[Caennanu]

Hmm, when you are playing and hosting at the same time, dropping game speed causes issues for other clients?
Never experienced that, than again i always seemed to be the slowest player.

i know loewchen submitted an idea that the gamespeed autoscales to the slowest player to prevent desync issues, not sure what the status is on that.
sorry mate, i think i ran out of idea's to help you out here.

[xBlizzDevious]

Hmm, when you are playing and hosting at the same time, dropping game speed causes issues for other clients?
Never experienced that, than again i always seemed to be the slowest player.

i know loewchen submitted an idea that the gamespeed autoscales to the slowest player to prevent desync issues, not sure what the status is on that.
sorry mate, i think i ran out of idea's to help you out here.

No, when hosting a dedicated server and a connecting admin runs game.speed it causes issues.

That desync prevention technique is what I was going to suggest. Run as slow as the slowest connected player and it should be good.

[xBlizzDevious]

Good news! I think I found the problem.

It's the one thing I imagined it was and really, really didn't want it to be.

Factorissimo2.

If I load up the save with all mods enabled in single-player, I get 60/60 FPS/UPS.
If I load up the save with all mods enabled in local multiplayer, I get 45-50/45-50.
If I load up the save with all mods EXCEPT Factorissimo2 in local multiplayer, I get 60/60.

Now there would be a difference in that there will be several hundred machines not running when Factorissimo2 is removed, but there is so much more outside of Factorissimo in this base (only smelting is done inside them) that it won't make that huge a difference.

[orzelek]

Good news! I think I found the problem.

It's the one thing I imagined it was and really, really didn't want it to be.

Factorissimo2.

If I load up the save with all mods enabled in single-player, I get 60/60 FPS/UPS.
If I load up the save with all mods enabled in local multiplayer, I get 45-50/45-50.
If I load up the save with all mods EXCEPT Factorissimo2 in local multiplayer, I get 60/60.

Now there would be a difference in that there will be several hundred machines not running when Factorissimo2 is removed, but there is so much more outside of Factorissimo in this base (only smelting is done inside them) that it won't make that huge a difference.

Did you look at the performance stats to see script usage for Factorissimo2 ?
I'm guessing it will depend on how many factory buildings you have but it will add up pretty quickly since scripts need to be used to transfer all the resources and power (and pollution out) for each factory building.

[xBlizzDevious]

Good news! I think I found the problem.

It's the one thing I imagined it was and really, really didn't want it to be.

Factorissimo2.

If I load up the save with all mods enabled in single-player, I get 60/60 FPS/UPS.
If I load up the save with all mods enabled in local multiplayer, I get 45-50/45-50.
If I load up the save with all mods EXCEPT Factorissimo2 in local multiplayer, I get 60/60.

Now there would be a difference in that there will be several hundred machines not running when Factorissimo2 is removed, but there is so much more outside of Factorissimo in this base (only smelting is done inside them) that it won't make that huge a difference.

Did you look at the performance stats to see script usage for Factorissimo2 ?
I'm guessing it will depend on how many factory buildings you have but it will add up pretty quickly since scripts need to be used to transfer all the resources and power (and pollution out) for each factory building.

I did. It was WaitForUpdate that kept going up. Factorissimi2 was only showing a couple of milliseconds which is why I was so baffled. But testing showed that removing it fixes it.

[orzelek]

Good news! I think I found the problem.

It's the one thing I imagined it was and really, really didn't want it to be.

Factorissimo2.

If I load up the save with all mods enabled in single-player, I get 60/60 FPS/UPS.
If I load up the save with all mods enabled in local multiplayer, I get 45-50/45-50.
If I load up the save with all mods EXCEPT Factorissimo2 in local multiplayer, I get 60/60.

Now there would be a difference in that there will be several hundred machines not running when Factorissimo2 is removed, but there is so much more outside of Factorissimo in this base (only smelting is done inside them) that it won't make that huge a difference.

Did you look at the performance stats to see script usage for Factorissimo2 ?
I'm guessing it will depend on how many factory buildings you have but it will add up pretty quickly since scripts need to be used to transfer all the resources and power (and pollution out) for each factory building.

I did. It was WaitForUpdate that kept going up. Factorissimi2 was only showing a couple of milliseconds which is why I was so baffled. But testing showed that removing it fixes it.

You could try to bug report it then.. there might be an issue with rendering when you have lots of factories.

[xBlizzDevious]

Good news! I think I found the problem.

It's the one thing I imagined it was and really, really didn't want it to be.

Factorissimo2.

If I load up the save with all mods enabled in single-player, I get 60/60 FPS/UPS.
If I load up the save with all mods enabled in local multiplayer, I get 45-50/45-50.
If I load up the save with all mods EXCEPT Factorissimo2 in local multiplayer, I get 60/60.

Now there would be a difference in that there will be several hundred machines not running when Factorissimo2 is removed, but there is so much more outside of Factorissimo in this base (only smelting is done inside them) that it won't make that huge a difference.

Did you look at the performance stats to see script usage for Factorissimo2 ?
I'm guessing it will depend on how many factory buildings you have but it will add up pretty quickly since scripts need to be used to transfer all the resources and power (and pollution out) for each factory building.

I did. It was WaitForUpdate that kept going up. Factorissimi2 was only showing a couple of milliseconds which is why I was so baffled. But testing showed that removing it fixes it.

You could try to bug report it then.. there might be an issue with rendering when you have lots of factories.

Three visible ones and about three inside each of those. That's all there is. You think it's worth reporting? And do you mean to Factorio or to the guy that made Factorissimo?

[orzelek]

If looking at stats game processing is below 13ms and UPS/FPS starts dropping then it might be something to investigate for devs.
Especially that it's a difference between single and multiplayer.

If detailed timings show that Factorissimo2 is using more script time in multiplayer then single that might be something for mod author to look at.

Both of those cases seem complicated

[xBlizzDevious]

If looking at stats game processing is below 13ms and UPS/FPS starts dropping then it might be something to investigate for devs.
Especially that it's a difference between single and multiplayer.

If detailed timings show that Factorissimo2 is using more script time in multiplayer then single that might be something for mod author to look at.

Both of those cases seem complicated

I would note that even on SP I'm running sub 60 FPS/UPS, hovering around 45-50. Local MP is about 30.
Not exactly scientific testing, but I opened the game up in local multiplayer, showed time-usage and then waited ten seconds and took a screenshot.
Then I saved, exited and loaded the game up as single player, waited ten seconds and took a screenshot. Here are the results:

SP:

Multiplayer

Multiplayer Time Usage.png (1.83 MiB) Viewed 11381 times

Local MP:

Single Player

Single Player Time Usage.png (1.65 MiB) Viewed 11381 times

Not exactly sure what I'm looking for, but I can see that there's a 30% drop in performance when you load MP.

[MagmaMcFry]

Okay guys, Factorissimo mod author here. I'd like to have a look at your map if you don't mind, because this definitely shouldn't be Factorissimo's fault. Please post your save and mods folder.

E: I've had a look at the vanilla-loaders code, it looks pretty inefficient. Are you using lots of loaders on trains? That might cause lag spikes whenever a train arrives or leaves.

[orzelek]

Stats show above 3ms for both Factorissimo2 and loaders.
I have no idea why loaders mod needs scripting and I think that bug it was working around was fixed in 0.15.x and it shouldn't need scripting at all.

[Nexela]

Loaders don't work on trains without scripting. 3ms for loaders means probably about 100+ loaders loading to/from trains

[xBlizzDevious]

Okay guys, Factorissimo mod author here. I'd like to have a look at your map if you don't mind, because this definitely shouldn't be Factorissimo's fault. Please post your save and mods folder.

E: I've had a look at the vanilla-loaders code, it looks pretty inefficient. Are you using lots of loaders on trains? That might cause lag spikes whenever a train arrives or leaves.

Stats show above 3ms for both Factorissimo2 and loaders.
I have no idea why loaders mod needs scripting and I think that bug it was working around was fixed in 0.15.x and it shouldn't need scripting at all.

To both of you: Loaders was added (and modified) since the issues with multiplayer lag started being a thing. Though my train stations are now epicly loader-heavy. 2-8-2 trains with 6 loaders from each wagon into 6 loaders that feed into a warehouse.


To MagmaMcFry: I've just uploaded the save and all mods to the bug reports forum as I was having another bug. viewtopic.php?f=7&t=47735&p=275371#p275371
You can use the files attached there to test. Simply load in single player, see how smooth it runs, then load in local multiplayer and see how much worse it runs. Remove Factorissimo2 and load local MP again and it's running much more smoothly. There is always the fact that there are several hundred furnaces running inside those factories that disappeared but I don't think they'd cause a 30% performance drop. Thanks for looking into it.