[1.1.74] Require() with relative path in migration script behaves differently in Tip & Tricks scenario
Posted: Wed Jan 04, 2023 3:43 pm
by robot256
I received a report against Cargo Ships 0.1.19 for a broken require() inside a migration script when opening a tutorial scenario from within another game (with a fairly large mod set). Log file is attached. It appears to affect only tutorials that are loaded from the Tips and Tricks library. Loading save files and tutorial scenarios from the menu applies the migration correctly, even when loading a 0.18 savegame. This behavior is consistent on Linux and WIndows.
In the file "__cargo-ships__/migrations/migrations-0-1-11.lua", there is a line requiring a file from a relative path:
When loading a save from the main menu, this migration correctly calls the file "__cargo-ships__/logic/oil_placement.lua". When loading a Tips & Tricks scenario, it searches for "__cargo-ships__/migrations/logic/oil_placement.lua", which does not exist. Changing the migration script code to an absolute path fixes the issue:
This is an issue with a relative path being resolved incorrectly, similar to 67032 and viewtopic.php?p=556401 , but I wanted to document it separately because it happens within the same mod and apparently only in Tips and Tricks scenarios.
The earlier thread about cross-mod compatibility gives a good reason to avoid relative paths anyways, so I am going to update everything to use absolute paths in the future anyways.
Re: [1.1.74] Require() with relative path in migration script behaves differently in Tip & Tricks scenario
Posted: Mon Jan 09, 2023 8:45 pm
by robot256
Small update: PI-C was investigating another tutorial migration crash and discovered that the tutorial scenario script calls the migration scripts within the Lua environment of the scenario itself, not within mod sandboxes. This is the root cause for why relative path resolutions are affected, but it also means that migrations involving the global variable table do not operate correctly. I'm not sure if this is a bug or if it was intentional.
PI-C's suggestion was to check the Lua environment variable and skip some or all of the migration script if it is not in a mod-specific sandbox. But we don't know if that will have other implications. The main game tutorial levels are not affected, seemingly because they load from the main menu like a normal save with a Lua sandbox for every mod. Can a dev explain why migration scripts are run this way in tips & tricks tutorial scenarios? Is there a different way to handle the change of Lua environment for them? How much mod code is actually utilized in tips & tricks tutorial scenarios?
Thanks in advance.
Re: [1.1.74] Require() with relative path in migration script behaves differently in Tip & Tricks scenario
Posted: Mon Jan 09, 2023 8:55 pm
by Rseding91
Checking the code: mod-specific scripts are not run at all in tips and tricks scenarios. As far as I can tell the only thing run is the lua defined by the simulation and migration.lua files.
Most likely the issue is that second part: mod migration.lua files were never intended to run in simulations.
Re: [1.1.74] Require() with relative path in migration script behaves differently in Tip & Tricks scenario
Checking the code: mod-specific scripts are not run at all in tips and tricks scenarios. As far as I can tell the only thing run is the lua defined by the simulation and migration.lua files.
Most likely the issue is that second part: mod migration.lua files were never intended to run in simulations.
OK, I've made a simple mod (mig-test_0.0.1.zip), started a new game with that, ran the stack-transfers tutorial, and saved the game:
0.000 2023-01-10 00:48:35; Factorio 1.1.74 (build 60256, linux64, full)
0.011 Operating system: Linux (Debian n/a)
0.011 Program arguments: "./factorio"
0.011 Config path: /home/pc/GOG_Games/Factorio/test_1.1_migration/config/config.ini
0.011 Read data path: /home/pc/GOG_Games/Factorio/test_1.1_migration/data
0.011 Write data path: /home/pc/GOG_Games/Factorio/test_1.1_migration [134115/1762036MB]
0.011 Binaries path: /home/pc/GOG_Games/Factorio/test_1.1_migration/bin
0.019 System info: [CPU: Intel(R) Core(TM) i5-2500 CPU @ 3.30GHz, 4 cores, RAM: 15946 MB]
0.019 Environment: DISPLAY=:0 WAYLAND_DISPLAY=<unset> DESKTOP_SESSION=lightdm-xsession XDG_SESSION_DESKTOP=lightdm-xsession XDG_CURRENT_DESKTOP=MATE __GL_FSAA_MODE=<unset> __GL_LOG_MAX_ANISO=<unset> __GL_SYNC_TO_VBLANK=<unset> __GL_SORT_FBCONFIGS=<unset> __GL_YIELD=<unset>
0.019 Display options: [FullScreen: 1] [VSync: 1] [UIScale: automatic (100.0%)] [Native DPI: 1] [Screen: 255] [Special: lmw] [Lang: en]
0.044 Available displays: 1
0.044 [0]: SMB2430L 24" - {[0,0], 1920x1080, SDL_PIXELFORMAT_RGB888, 60Hz}
0.127 Initialised OpenGL:[0] NVIDIA GeForce GTX 1050/PCIe/SSE2; driver: 3.3.0 NVIDIA 510.108.03
0.127 [Extensions] s3tc:yes; KHR_debug:yes; ARB_clear_texture:yes, ARB_copy_image:yes
0.127 [Version] 3.3
0.127 Verbose GraphicsInterfaceOpenGL.cpp:947: [Caps] Tex:32768, TexArr:2048, TexBufSz:131072kB; TexUnits:192; UboSz:64kB
0.127 Graphics settings preset: high
0.127 Dedicated video memory size 2048 MB
0.145 Verbose PipelineStateObject.cpp:85: Time to load shaders: 0.017529 seconds.
0.151 Graphics options: [Graphics quality: normal] [Video memory usage: high] [DXT: low-quality] [Color: 16bit]
0.151 [Max threads (load/render): 32/4] [Max texture size: 0] [Tex.Stream.: 1] [Rotation quality: low] [Other: STDCwt] [B:0,C:0,S:100]
0.154 [Audio] Backend:SDL(default); Driver:pulseaudio, Device:Default device, Depth:16, Channels:2, Frequency:44100
0.236 Loading mod core 0.0.0 (data.lua)
0.284 Loading mod base 1.1.74 (data.lua)
0.489 Loading mod base 1.1.74 (data-updates.lua)
0.582 Checksum for core: 870127790
0.582 Checksum of base: 4211423066
0.582 Checksum of mig-test: 0
0.801 Finished checking unused prototype data in 0.002573 seconds. Number of properties that were used: 158763
0.808 Prototype list checksum: 3034860339
0.813 Verbose ModManager.cpp:603: Time to load mods: 0.580553
0.853 Loading sounds...
0.889 Info PlayerData.cpp:71: Local player-data.json available, timestamp 1673308083
0.890 Info PlayerData.cpp:78: Cloud player-data.json unavailable
1.031 Initial atlas bitmap size is 16384
1.034 Created atlas bitmap 16384x6272 [none]
1.034 Created atlas bitmap 4096x2120 [decal]
1.034 Created atlas bitmap 8192x2928 [low-object, linear-minification]
1.035 Created atlas bitmap 4096x2464 [mipmap, linear-minification, linear-magnification, linear-mip-level]
1.035 Created atlas bitmap 8192x3840 [terrain, mipmap, linear-minification, linear-mip-level]
1.035 Created atlas bitmap 4096x1632 [terrain-effect-map, mipmap, linear-minification, linear-mip-level]
1.036 Created atlas bitmap 4096x1664 [smoke, mipmap, linear-minification, linear-magnification]
1.036 Created atlas bitmap 4096x928 [mipmap]
1.036 Created atlas bitmap 8192x2240 [icon, not-compressed, mipmap, linear-minification, linear-magnification, linear-mip-level]
1.036 Created atlas bitmap 4096x828 [alpha-mask]
1.039 Created atlas bitmap 16384x3820 [shadow, linear-magnification, alpha-mask]
1.039 Created atlas bitmap 4096x1312 [shadow, mipmap, linear-magnification, alpha-mask]
1.039 Created atlas bitmap 2048x240 [icon-background, not-compressed, mipmap, linear-minification, linear-magnification, linear-mip-level, ]
1.049 Verbose AtlasSystem.cpp:283: Created virtual atlas page 4096x4096 [none]
1.052 Verbose AtlasSystem.cpp:283: Created virtual atlas page 4096x3584 [linear-magnification]
1.053 Created virtual atlas pages 4096x4096x2
1.064 Verbose TextureProcessor.cpp:1136: GPU accelerated compression test: Time 11.035 ms; YCoCg_DXT PSNR: 35.83, BC3 PSNR: 33.82
1.064 Texture processor created (2048). GPU accelerated compression Supported: yes, Enabled: yes/yes. Test passed. YCoCgDXT PSNR: 35.83, BC3 PSNR: 33.82
1.313 Parallel Sprite Loader initialized (threads: 3)
6.168 Sprites loaded
6.263 Generated mipmaps (3) for atlas [2] of size 8192x2928
6.286 Generated mipmaps (3) for atlas [3] of size 4096x2464
6.355 Generated mipmaps (3) for atlas [4] of size 8192x3840
6.368 Generated mipmaps (3) for atlas [5] of size 4096x1632
6.383 Generated mipmaps (3) for atlas [6] of size 4096x1664
6.411 Generated mipmaps (3) for atlas [7] of size 4096x928
6.481 Generated mipmaps (5) for atlas [8] of size 8192x2240
6.489 Generated mipmaps (3) for atlas [11] of size 4096x1312
6.536 Custom mipmaps uploaded.
6.615 Generated mipmaps for virtual atlas of size 4096x8192
6.615 Verbose AtlasSystem.cpp:952: Virtual atlas mipmaps generated in 0.079147 seconds.
6.615 Verbose AtlasSystem.cpp:955: Atlas memory size: 286.59MB; 39.38MB (virtual)
6.615 Verbose AtlasSystem.cpp:956: Size of sprites outside of atlas: 0.13MB
6.693 Factorio initialised
6.695 Info HttpSharedState.cpp:54: Downloading https://updater.factorio.com/updater/get-available-versions?username=Pi-C&token=<private>&apiVersion=2
7.402 0 packages available to download (experimental updates enabled).
29.841 Loading map /home/pc/GOG_Games/Factorio/test_1.1_migration/saves/migration_test_tutorial.zip: 954038 bytes.
29.856 Loading level.dat: 1106945 bytes.
29.860 Info Scenario.cpp:199: Map version 1.1.74-1
29.918 Verbose Scenario.cpp:255: Loading level.dat finished: 0.058535 seconds.
29.919 Verbose BlueprintLibrary.cpp:67: Loaded external blueprint storage: playerIndex=0, nextRecordID=0, timestamp=1673307417, records={}
29.920 Verbose Scenario.cpp:339: Entities setup finished: 0.001442 seconds.
29.921 Loading script.dat: 1220 bytes.
29.924 Checksum for script /home/pc/GOG_Games/Factorio/test_1.1_migration/temp/currently-playing/control.lua: 2881393120
29.924 Checksum for script __mig-test__/control.lua: 502127159
29.930 Applying migration: Migration test: test_1.lua
29.930 Script @__mig-test__/migrations/test_1.lua:1: script.mod_name: mig-test
29.930 Script @__mig-test__/migrations/test_1.lua:3: global: {
test = "TEST"
}
29.930 Script @__mig-test__/migrations/test_1.lua:6: Defined CHECK_X: TESTSTRING
29.930 Applying migration: Migration test: test_2.lua
29.930 Script @__mig-test__/migrations/test_2.lua:1: script.mod_name: mig-test
29.930 Script @__mig-test__/migrations/test_2.lua:2: global: {
test = "TEST"
}
29.930 Script @__mig-test__/migrations/test_2.lua:4: CHECK_X: TESTSTRING
29.931 Info LuaGameScript.cpp:790: level state size: 1157 bytes.
29.931 Info LuaGameScript.cpp:790: level state size: 1157 bytes.
29.931 Script @__mig-test__/control.lua:7: global (on_config_changed): {
test = "TEST"
}
29.931 Verbose Scenario.cpp:389: Map setup finished: 0.089384 seconds.
As you can see, both migration scripts show the global table I've set up in on_init. Also, the global variable CHECK_X which I have defined in migrations/test_1.lua still has the same value in migrations/test_2.lua. You can also see that script.mod_name is the name of my mod.
79.941 Supending game state
79.941 Entering tutorial: [base]stack-transfers
79.957 Loading level.dat: 1185316 bytes.
79.957 Info Scenario.cpp:199: Map version 0.18.1-0
79.959 Info PrototypeMigrationList.cpp:194: Activating migration base/1.1.0.json
80.024 Verbose Scenario.cpp:255: Loading blueprint.dat finished: 0.066759 seconds.
80.024 Verbose BlueprintLibrary.cpp:67: Loaded external blueprint storage: playerIndex=0, nextRecordID=0, timestamp=1673307417, records={}
80.025 Verbose Map.cpp:1181: Time to re-chart all: 0.000049 seconds.
80.025 Verbose Scenario.cpp:339: Entities setup finished: 0.001576 seconds.
80.028 Checksum for script /home/pc/GOG_Games/Factorio/test_1.1_migration/temp/currently-playing-tutorial/control.lua: 939336307
80.029 Applying migration: Migration test: test_1.lua
80.030 Script @__mig-test__/migrations/test_1.lua:1: script.mod_name: level
80.030 Script @__mig-test__/migrations/test_1.lua:3: global: {}
80.030 Script @__mig-test__/migrations/test_1.lua:6: Defined CHECK_X: TESTSTRING
80.030 Applying migration: Migration test: test_2.lua
80.031 Script @__mig-test__/migrations/test_2.lua:1: script.mod_name: level
80.031 Script @__mig-test__/migrations/test_2.lua:2: global: {}
80.031 Script @__mig-test__/migrations/test_2.lua:4: CHECK_X: nil
80.037 Info LuaGameScript.cpp:790: level state size: 270 bytes.
80.040 Checksum for script /home/pc/GOG_Games/Factorio/test_1.1_migration/temp/currently-playing-tutorial/control.lua: 939336307
80.040 Verbose Scenario.cpp:389: Map setup finished: 0.083613 seconds.
80.041 Info LuaGameScript.cpp:790: level state size: 270 bytes.
My migration scripts are run again, but this time, script.mod_name is "level" and the table "global" is empty. What's even more interesting is that my definition of the global variable CHECK_X from migrations/test_1.lua has been invalidated in migrations/test_2.lua.
Re: [1.1.74] Require() with relative path in migration script behaves differently in Tip & Tricks scenario
Most likely the issue is that second part: mod migration.lua files were never intended to run in simulations.
To clarify:
Do you mean that it was not intended for the simulation script to call mod migration.lua files, and the simulation script needs to be fixed by you to not call them anymore?
Or do you mean that the mod migration.lua files were written without the intention they would be run in a simulation, and we need to fix them to account for this situation?
Re: [1.1.74] Require() with relative path in migration script behaves differently in Tip & Tricks scenario
Do you mean that it was not intended for the simulation script to call mod migration.lua files, and the simulation script needs to be fixed by you to not call them anymore?
This one.
Re: [1.1.74] Require() with relative path in migration script behaves differently in Tip & Tricks scenario