[0.15.35] Crashes and bad perf previewing/working larger BPs

[Qon]

So I tried tried to preview one of my larger blueprints by hovering my mouse over the blueprint item in my inventory and the game just froze and crashed. Updated to latest factorio version (0.15.35) and tried again with same results.

Reproduce:
Load up save file.
Press E to enter inventory.
Hover mouse over the blueprints in the inventory. The largest one is causing the crashes (Highlighted in the screenshot, to the left of the deconstructor, named "1024 Balancer")

I only have 8 GB of RAM on this machine. The blueprint string is only 9 MB however.

Pull yourselves together Wube

The SS shows a folder named factorio 0.15.30 but that is just folder name, not factorio version

ousu(170928-172602-86).jpg (1.13 MiB) Viewed 2811 times

Ok, since I don't want you to just dismiss it because of the sillyness... I actually think the blueprint previews, contruction, destruction etc does need improvements. Does the whole blueprint need to be expanded into memory completely to place/decon planned? No. And the preview doesn't need to render the blueprint at the same FPS as the rest of the world. The blueprint rendering of the blueprints entities can be completely asynchronous to the rest of the preview rendering that shows item requirement list and buttons (when you click it).

If I can build it manually and the game can handle it, then I should be able to blueprint it too. If I can manually work around the limitations of the current blueprint system by splitting large constructions into several smaller blueprints and place them sequentially then the game should be able to do that for me. Chunkify the blueprints and work with each segment sequentially.

The game easily supports large worlds with hundreds of thousands of entities, but the blueprint system is extremely inefficient in comparison. Would be nice if the games support for large blueprints were improved. I've noticed bad performance with smaller blueprints than that one where the game freezes and drops to sub 1 fps when previewing. And those blueprints that can exist as physical constructions of the entities in the world without problems.

Edit: The crashlog and savefile, right... http://www55.zippyshare.com/v/yItiAsSW/file.html

[Rseding91]

The reason you're getting such poor performance is due to the Creative Mode mod you're using. It changes the normal limits the game has to work within to crazy numbers to do things like the 15x15 chunk electric pole. It'a also responsible for increased load times on any save file you use it with.

[Qon]

The reason you're getting such poor performance is due to the Creative Mode mod you're using. It changes the normal limits the game has to work within to crazy numbers to do things like the 15x15 chunk electric pole. It'a also responsible for increased load times on any save file you use it with.

Rseding91...


Is that true even when no CM items are placed in the world? Did you actually investigate? I don't have full insight into the inner workings of Factorio, but your claim was a bit suspicius. So I am checking if your claim that it is due to mods might have some truth to it. I am basically only using CM for runspeed and instant blueprints and the only entity from CM that exists in the world during the crash is the smaller super-radar.

So what you said was a little bit true. Without any mods and on a completely new world it actually managed to render the preview of the 1024-balancer for 1 frame before it crashed (and taking firefox down with it) [factorio-current-nomods.log].
So I tried again with firefox off for some extra RAM available to Factorio. And it managed to render the preview without crashing, although the whole game was again at sub 1 fps/ups. So i picked up the blueprint to place it down and the game crashed again when the cursor went over the ground [factorio-current-nomods-nofirefox.log]. And I didnt even get a frame of the in-world-preview and I didn't actually click to place it down.
This time it took the battle.net launcher with it...

So without mods it's slightly better, but not so much better that it actually matters. The game still keeps crashing from blueprints that I could build manually.

[Rseding91]

You don't have to have 1 creative-mode item in the world for it to have the performance impact. That's why it's particularly bad. I also said nothing about the crash - that's unrelated to the performance issues.

[daniel34]

I tested it and it did not crash, but used up to 10.1 GB of RAM on that map. It probably crashes if you have less, but that's not (directly) Factorio's fault. It's not related to mods either - all my analysis was done on vanilla.

The performance was very, very low while handling the 1024-belt-balancer (898k underground belts, 159k transport belts, 34k splitters). The framerate was very low (<0.5FPS) while opening the blueprint/seeing the preview, but also while placing it down. After placing it down the framerate returned to 60. All of the low framerate time was spent in "Gui render preparation" while seeing the preview and "Game render preparation" while placing it down. Note that its not the GPU that does the preparation and caused the lag, its the CPU.

It makes sense that when looking at the blueprint preview the game has to include all the items in the blueprint and render a proper preview, but does it have to do that every tick? Blueprints are not inherently mutable, surely you could have a client database of blueprint pictures indexed by string/CRC instead of re-rendering it 60 times per second, or attach it to the blueprint entity (in memory) itself.

Then I tried to place the blueprint, but the same lag happened. The debug shows "Game render preparation", but there is only a small part of the blueprint displayed in the game window, most of it is off-screen. The game should only try to render the on-screen part and ignore the other parts. After placing the blueprint (laying down ~1 million ghosts, which actually doesn't take any noticeable time) and Q-ing the blueprint the game is back to 60 FPS/UPS.

So we can, apparently, lay down large blueprints and have no issues afterwards, but we have to endure massive lag while selecting and then placing the blueprint.

[Rseding91]

I looked at the crash and it's crashing because it's running out of RAM trying to render the blueprint preview.

Just loading the save takes 5.4 GB of RAM, if you also have the large blueprint copied into the library that adds another 1.5 GB and rendering the preview takes between 1 and 2 GB. It fails to allocate enough memory for the render preparation data because there's not a large enough contiguous piece of memory available for the request and it crashes.

I found one small area to improve memory usage in rendering the blueprint but the real "fix" if you don't want it to crash is either get more RAM or don't make blueprints that large

You'll get the same performance if you zoomed the normal game view out that far - simply we don't allow it in the normal world view because no computer that exists today can handle it without severe performance drops. When you put that much stuff into a blueprint it still tries to render it but you now get to experience the performance hit of doing so.

[Qon]

You don't have to have 1 creative-mode item in the world for it to have the performance impact. That's why it's particularly bad. I also said nothing about the crash - that's unrelated to the performance issues.

Maybe I wasn't clear. I'm only getting poor performance when activly dealing with blueprints, with Creative Mode Installed or without it. The CM impact, without spawning any of it's entities in the world, is negligable.

These two statements prove that mods are unrelated to my crash/performance report:

• The performance is fine as long as I don't try to use blueprints even with mods like Creative Mode.
• The the ups drops and the game crashes without mods when activly using large blueprints.

The performance issues and the crashes happen at the same stresspoints. The issues are linked. When I disabled CM I got marginally better performance and then could preview the 1024-balancer for a few frames (in certain circumstances) without crashing for a short while. Basically the game crashes if a frame hasn't been drawn for X amount of seconds (I'm not saying the frame delay literally causes the crash though).

Basically if the blueprint string was read in chunks and ghost were built 10k/frame max then the game would probably not crash at any blueprint size that your compter can run. The issue is that blueprints are massivly more performance intensive than actual entities.

It might even be possible to make a blueprint mod in lua that performs better (doesn't freeze or crash the game by doing updates over time) than the in-game blueprint system when dealing with large blueprints. Can't remember on the top of my head if the Lua-API gives me access to file reading which would make it fairly easy. Back to the old times when blueprints were a mod implementation. The C++ in-engine implementation shouldn't ever be beaten by a lua mod though if it is designed with large blueprints in mind. I'm not saying that it has to be solved exactly by adding ghosts over several frames, but it is a solution if you can't come up with something better.

____________
So after writing that above I see new responses that I will adress:

I tested it and it did not crash, but used up to 10.1 GB of RAM on that map. It probably crashes if you have less, but that's not (directly) Factorio's fault. It's not related to mods either - all my analysis was done on vanilla.

Yes, I assumed that was the case. How much memory did the game use after you had finished placing the ghost (assuming the game does free up unused memory)? If we relaxed the requirement to place everything at once, even for blueprints with 10^6 entities, we could reduce the memory requirement to something a bit lower, yes?

The performance was very, very low while handling the 1024-belt-balancer (898k underground belts, 159k transport belts, 34k splitters). The framerate was very low (<0.5FPS) while opening the blueprint/seeing the preview, but also while placing it down. After placing it down the framerate returned to 60. All of the low framerate time was spent in "Gui render preparation" while seeing the preview and "Game render preparation" while placing it down. Note that its not the GPU that does the preparation and caused the lag, its the CPU.

Yeah those slowdowns make working with large blueprints very, very painful. I can deal with placing a blueprint down and having the actual construction being completely built minutes later (instant blueprint, with robots it would take days though but maybe a minute to get the ghosts down if optimised). But playing the game at 0 FPS as soon as your cursor just hovers over an icon or the entire time while positioning the blueprint is quite bad.

It makes sense that when looking at the blueprint preview the game has to include all the items in the blueprint and render a proper preview, but does it have to do that every tick? Blueprints are not inherently mutable, surely you could have a client database of blueprint pictures indexed by string/CRC instead of re-rendering it 60 times per second, or attach it to the blueprint entity (in memory) itself.

Then I tried to place the blueprint, but the same lag happened. The debug shows "Game render preparation", but there is only a small part of the blueprint displayed in the game window, most of it is off-screen. The game should only try to render the on-screen part and ignore the other parts. After placing the blueprint (laying down ~1 million ghosts, which actually doesn't take any noticeable time) and Q-ing the blueprint the game is back to 60 FPS/UPS.

I think your optimisation ideas would drastically enhance the usability of using large blueprints.

So we can, apparently, lay down large blueprints and have no issues afterwards, but we have to endure massive lag while selecting and then placing the blueprint.

Yep, exactly.
___________________________________________

I looked at the crash and it's crashing because it's running out of RAM trying to render the blueprint preview.

Not that surprising to me since I could barely generate the blueprint on my computer without running out of RAM and since the game used up all of it trying to work wih the blueprint. But the blueprint library tries to do the simple and easy process of keeping it all in memory at once when. The game should be smarter about it

Just loading the save takes 5.4 GB of RAM, if you also have the large blueprint copied into the library that adds another 1.5 GB and rendering the preview takes between 1 and 2 GB. It fails to allocate enough memory for the render preparation data because there's not a large enough contiguous piece of memory available for the request and it crashes.

You'll get the same performance if you zoomed the normal game view out that far - simply we don't allow it in the normal world view because no computer that exists today can handle it without severe performance drops. When you put that much stuff into a blueprint it still tries to render it but you now get to experience the performance hit of doing so.

I placed the blueprint in the library to be able to hover over it to get the preview. Would it take less memory to place it in my regular inventory? I did notice that the game froze for a few seconds moving the item from my cursor to the game library.

With daniels suggestions, why even render the whole item at once? It's never displayed at zoom level 1 and the belt items aren't connected in the blueprint view. The image database could also be built up asyncronously over time. The preview window could then pop up immediatly showing item requirement list and so on while the render preview is updated piecewise while the game plays on at a nice and smooth 60 fps with low RAM requirements. The render prep wouldn't be so big if you never try to prepare to render all of it in one go.

You'll get the same performance if you zoomed the normal game view out that far - simply we don't allow it in the normal world view because no computer that exists today can handle it without severe performance drops. When you put that much stuff into a blueprint it still tries to render it but you now get to experience the performance hit of doing so.

But if I did that I would actually want the game to render the whole view each frame. For a static blueprint shown a low resolution with low requirements on update speed and acceptable delay for the entity rendering part of the preview it's not at all the same thing. The tasks are completely different from a user perspective and that means relaxed requirements from a technical perspective. One shows a picture and another shows an interactive game.

I found one small area to improve memory usage in rendering the blueprint but the real "fix" if you don't want it to crash is either get more RAM or don't make blueprints that large

Hey, I'm not giving up and daniel isn't giving up, so you can't either!
Did you forget that making oversized contruction is the point of Factorio? And you are the legendary optimizer on Wube destined to save us all? What about your reputation, the glory and half the kingdom?

[Jap2.0]

Sorry if I don't understand the engine, but couldn't the preview on the ground (not when you mouse over/open the blueprint) be set so that it only renders items within either a few chunks of the player (ex. whatever the maximum view distance is) or just the chunks the player can see (be it in the map or where the player is standing)?

[eradicator]

Just loading the save takes 5.4 GB of RAM, if you also have the large blueprint copied into the library that adds another 1.5 GB and rendering the preview takes between 1 and 2 GB.

How does this interact with multiplayer blueprint sharing? If i theoretically had 10 such blueprints in my library, would everyone with less than 16+ gigs of ram just crash if i join a game? And why does it require that much space for just being there. Is it not just a stored string/table?

Also honestly i've always wished for a zoom function for very large blueprints. Previewing that many entities in an - in comparison - tiny window is often not very useful. And hopefully with default ~= max zoom you could then save on rendering the whole thing at once.