[0.11.16] [cube] import "large" blueprints -> computer locks up

[NotABiter]

I am using the Blueprint String mod (version 0.0.6) to load blueprints into the game. If I load a blueprint which contains an array of 32x32 requester chests (1024 total chests), it works fine. If I try to load a blueprint which contains an array of 64x32 requester chests (2048 total chests), the game freezes, alt-tab and ctrl-shift-esc soon stop working, eventually even basic keyboard processing stops working (e.g., clicking num-lock doesn't toggle the num-lock light) and I end up with a completely black screen. At that point reset/power buttons are the only way to bring my computer back to life.

I am running on a 4GB machine with pretty much nothing else running, so about 3GB of available memory. On one attempt I managed to get to TaskManager before the computer become completely nonresponsive, and what I saw was memory ("Commit Size" for Factorio.exe) shooting up very quickly through 2GB and reaching 3,392,828K before TaskManager finally froze. This was done immediately after loading a save-game, so without trying to load the blueprint I loaded that save game and looked in TaskManager and Factorio.exe was only using 1,215,540K. By my calculation, somehow it would appear that more than (3,392,828K - 1,215,540K) / 2048 or about 1MB of memory is needed for each requester chest?

I also tried a fresh start of the game, loading the same save game, and then loading the 32x32 array to see how much memory usage increased in that case, and I "only" saw it go up by 39,300K. (That's a bit over 38K per requester chest, which seems a lot more reasonable.)

The freezing behavior also seems odd. I have the max page file size set to ~8GB, and there's 34GB free on that drive, so (just as has happened to me in the past many times when I've used too much memory) I would expect a slow down (until I shut down or killed whatever was hogging memory). Though I think I've only experienced that with regular windows apps (e.g., IE/FireFox) sucking up memory - not games, and maybe windows (Vista 64 SP2) isn't so stable when it's a DirectX-app/game hogging so much memory. (I am running Factorio full screen.)

I have zipped and attached the blueprints. 32x32_blueprint works, while 64x32_blueprint causes failure.

I also tried the following: I applied the 32x32_blueprint twice in the game to set up a 64x32 array of ghost logistic chests, then blueprinted that in game, exported that blueprint using the "blueprint string" mod, and then tried to reimport the exported blueprint. That failed just like trying to import my original 64x32_blueprint does.

There's nothing interesting in the log:

Code: Select all

52852.607713  Info Logger.cpp:152: 2015-03-01 21:27:58; Factorio 0.11.16 (Build 13544, win64)
52852.607786  Info Logger.cpp:156: Operating system: Windows Vista Service Pack 2
52852.607821  Info Paths.cpp:202: Read data path: C:/games/factorio/data
52852.607854  Info Paths.cpp:203: Write data path: C:/games/factorio
52852.607886  Info Paths.cpp:204: Binaries path: C:/games/factorio/bin
52853.318448  Info ModManager.cpp:205: Loading mod core 0.0.0 (data.lua)
52853.359923  Info ModManager.cpp:205: Loading mod base 0.11.16 (data.lua)
52853.988118  Info ModManager.cpp:205: Loading mod base 0.11.16 (data-updates.lua)
52855.244702  Info AtlasSystem.cpp:157: Initial atlas bitmap size is 8192
52855.260387  Info AtlasSystem.cpp:91: Created atlas bitmap 8192x8187
52855.683265  Info AtlasSystem.cpp:91: Created atlas bitmap 8192x4935
52889.686495  Notice Scenario.cpp:125: Loading map C:/games\factorio\saves\comp014.zip
52890.018542  Info Scenario.cpp:158: Map version 0.11.15-0

[Rseding91]

This doesn't actually have anything to do with the "setblueprintentities" method but the fact the blueprint string mod has stripped the [#] tags off each entry. Some part of Factorio's internal Lua handling doesn't know how to process the table of tables correctly when they're not explicitly named.

For Cube:

I think this has to do with the determistic iteration changes to the Lua internals. If you try take a table of 1024 auto-named entries it iterates just fine but if you make it 1025 it freezes and goes into some type of infinite loop.

auto-named entries being: {{one = "test"}, {two = "test again"}}

Instead of: {[1]={one = "test"}, [2]={two = "test again"}}

[DaveMcW]

Freezes from the command line too.

Code: Select all

/c t=0; for _,n in pairs{
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
} do t=t+n end; game.player.print(t)

[cube]

Fixed for 0.11.17. DaveMcW, thanks for the example a little more compressed version went directly into our unit tests :-).

[DaveMcW]

This still happens in 0.11.17, the freeze point is tables of size 1101 or larger.

Here is a more robust test case for you.

Code: Select all

/c size = 100000;
array = loadstring("return {1"..(",1"):rep(size-1).."}")();
total = 0;
for _,n in pairs(array) do total = total + n end;
game.player.print(total)

[DaveMcW]

If increasing the memory limit for pairs() is too hard, I would be happy if you replaced pairs() with ipairs() in setblueprintentities().

[cube]

Fixed lockup when iterating large numerical arrays in lua. This time for real. (https://forums.factorio.com/forum/vie ... 19&p=71104)

... everything over 1024 uses the same code path and can't fail if 1025 has been tested? Yeah, but many parts of lua use many different tables with many different sizes. Oh joys of programming. This time I tested up to 100000 :-)