[boskid][2.0.5] Running JSON migrations with Lua code is painfully slow

[Atria]

For example when I have SA with just one mods (attached) which have this migration code, which does actually no migration:

Code: Select all

{
    "item": [
        "for k, v in pairs(mapping) do end"
    ],
    "recipe": [
        "for k, v in pairs(mapping) do end"
    ],
    "entity": [
        "for k, v in pairs(mapping) do end"
    ]
}

just loading of menu simulations can take anywhere from a second to 10 seconds (depends on which simulation is loading).

The issue gets worse the more mods are using the code migrations (even if they are "empty" in same way). Enabling all three attached mods can freeze loading of menu simulations for up to 30 seconds from my experience.

It gets even worse when those migration scripts are actually doing some stuff (like string.gsub) and when loading larger maps. You can easily look at 10-20 minutes of loading with 3 mods.

[boskid]

I did a quick measurement using 36 of such lua json migrations, and background simulations were taking about 30 seconds to load each... it looks like populating script state with a copy of data.raw is expensive, and migrations are running when loading every single blueprint separately inside of a save file (each blueprint may have different ID mapping and need different migrations to be applied) a decision was made that this feature will get removed. I will mark this topic as resolved once this feature is fully removed.

[boskid]

Thanks for the report. Unfortunately dynamic migrations (lua code in json) will get removed in 2.0.7.

[Atria]

Would it be possible to still allow to run Lua script to generate array of normal migrations instead of listing them manually?

[boskid]

I have an idea how this could be done in a performance friendly way which does not involve putting lua code inside of json files but this is low priority thing right now and it can be done after release when there will be calm period.