Mod support: reload data.raw related mods

[SirRichie]

This idea is about improving the usability for mods.

When you create/use a mod that makes changes to data.raw (the things you'd typically place in a data.lua or data-updates.lua), you need to restart the game for the mod to take effect. This is somewhat cumbersome when you develop the mod, but also when you use such mods. For example, you have savegames with different mods, or the multiplayer server you want to join has a different set of mods enabled.

Having some way to reload the data-definitions, without restarting Factorio would be very convenient in these cases. You would simply need to have the mods a multiplayer server is using, and not need the same enalbed set.

I imagine that this requires some heavy changes to the loading routines, but I would appreciate it nonetheless.

TL;DR
A way to reload data-definitions without restarting Factorio would be very convenient.

[Rseding91]

This idea is about improving the usability for mods.

When you create/use a mod that makes changes to data.raw (the things you'd typically place in a data.lua or data-updates.lua), you need to restart the game for the mod to take effect. This is somewhat cumbersome when you develop the mod, but also when you use such mods. For example, you have savegames with different mods, or the multiplayer server you want to join has a different set of mods enabled.

Having some way to reload the data-definitions, without restarting Factorio would be very convenient in these cases. You would simply need to have the mods a multiplayer server is using, and not need the same enalbed set.

I imagine that this requires some heavy changes to the loading routines, but I would appreciate it nonetheless.

TL;DR
A way to reload data-definitions without restarting Factorio would be very convenient.

Reloading data.lua (and the other staged variants) requires that sprites be re-loaded - that's 95% of the loading time when the game is restarted. What might be nice is a "restart game" button that closes the game and re-loads it.

[ssilk]

A good question is here: When the sprites don't change, why reload it? I mean it's a good question to ask, cause I think many just want change the code a bit and now need to reload only the code, not all.

[SirRichie]

Well, anything script-related is reloaded when the map is loaded/generated, so that is fine. But anything that affects data.raw is not. For example, you have been playing with mods A, B and C but now want to join a friend's server, with a savegame that requires mods A,B and D. If D does changes to data.raw, you will have to activate it (and deactivate C) and then restart Factorio.

I think it is more convenient (and that is all this request is about) to have the game detect the required mod configuration and automatically reload entity definitions (it's not just sprites) in the background.

[SirRichie]

As an additional note: sprites/data could be loaded lazily, i.e. not at startup, but at the initialization of the actual game. The total loading time would be the same. If the selected mods do not change, she a second game can use the already loaded data. However, if the active mods did change, sprites can be reloaded.
But my concern is not with minimizing loading times, it is with convenience.

[Rseding91]

Don't forget that the data.raw contents are determistically created based off the mods and the mod load order. If you loaded with mods A, B, C and D and then removed B it would require you re-load data.raw starting at A, then C then D. You couldn't simply say "remove B" and know for sure that B didn't alter A and that C and D didn't alter the alterations of B or contents of B thus changing what they would have done during load.

[SirRichie]

I am very much aware of that. What I am suggesting is basically to move the data.raw loading process from startup to right before game initialization/load.
Thereby, the process itself can remain the same. As an optimization, if mods did not change and data.raw was loaded before, then there is no need to re-load it.
As a further optimization, this loading could be done in the background (i.e. on a second core while the user is clicking through the UI).

[bobingabout]

"Lazy loading" could be beneficial in some places, but I think the best solution would be to fully reload the data.raw, which should write a table of sprites to load into GRAM, then when the data.raw has finished loading, it compares this newly generated table with the table of existing sprites, and loads any that are missing, or have changed. it could possibly even unload sprites that are no longer in this new list. This could drastically save some loading times.

Though there are issues when loading new mods, for example, mods are only checked for viabillity and compatabillity when loaded, unloaded mods are not checked, it may be possible that a new config could cause errors, in this case, should the game close as it does when loading the game with broken mods now?

[SirRichie]

I think we need to distinguish between optimizing the loading itself (which is what you suggest) and the question about when the loading happens (which is what I suggest). These ideas do not conflict and could be implemented together (in fact, they profit from each other).

The concern about invalid mods is a good one. I think that any invalid mods -upon activation- should just be disabled again, with a warning message that the mod could not be loaded. So basically undo whatever the loading process did until this point.
Of course when loading a save that requires a certain mod, which has become invalid, this leads to a chain of consequences (i.e. game cannot be loaded), but I think anything is better than force-closing the game.

[bobingabout]

Take my previous post as "This is what should happen when you exit the mod options page". Or perhaps press a new "Apply config" button within the page.

[orzelek]

I think that "lazy loading" might not be well suited for Factorio.
From what I noticed game uses a big dictionary and tries to fit it as big textures as driver allows. This basically means that any update would cause a texture update for currently rendered one which is not something you want to do. Also building a dictionary would be potentially more complex if it would need to reserve space for further pictures or try to distribute stuff around.

I do agree that a mod reload button would be very useful. For each small tweak we need to reload the game now (at least with zipped mods - haven't tried after unzipping but zip files are locked when the game is running).

[SirRichie]

I fail to see how the work that happens during loading has anything to do with how suitable "lazy loading" is.

I'll try to illustrate this, by showing a simplified process of what Factorio has to do for the player to start a game:

Code: Select all

load game data (may include complex operations) -- load UI -- wait for user input -- generate world -- run game

Now, what I am suggesting is that we move the loading process:

Code: Select all

load UI -- wait for user input -- load game data -- generate world -- run game

Obviously, that is less favorable, because now we have to load the game data (which is a time-consuming task) every time we start a game, so we should change that a little bit:

Code: Select all

load UI -- load game data -- wait for user input -- check if game data changed (i.e. mod update / mod activation), and if so, load game data (again) -- generate world -- run game

So now the process is very similar to Factorio's current process, except that we have a small, very inexpensive check. In fact, using this process, you can restart Factorio after activating mods and you will not notice a difference to how it is done at the moment.
However, if I do not restart, Factorio takes care of updating my game data, which of course takes quite some time, but that is time I would have had to spend anyway.

There is, however, even more optimization possible (as suggested in the above responses):
Factorio can potentially load the game data in the background while waiting for user input. In addition, it can optimize the process of updating/reloading the game data, by loading only those things that have changed.

Another benefit with my suggestion is that given a savegame that requires a different mod configuration than the current one. You would no longer have to check the required mods of the savegame, then switch to that config, restart and then load. With my suggestion, you could simply load the savegame and Factorio would reload the game data according to the required mods. Of course that means the loading time increases significantly (though again the total amount of work remains the same), but I would not have to worry about mod configuration myself.

[bobingabout]

Well, the data will only ever change if you change the mods, and this is done in the mod config screen, which is why I sugested that it reloads data when you exit the mod config screen, if the config changes.

I expanded on that to say that it should only reload sprite sheets if they have changed, because a huge ammount of the loading time is taken up by loading sprites.

[vzybilly]

just adding in my thoughts about a way to ease this, just load all sprites even if a mod doesn't use them, allowing the data step to run more dynamically...

[SirRichie]

just adding in my thoughts about a way to ease this, just load all sprites even if a mod doesn't use them, allowing the data step to run more dynamically...

So that would mean increasing the initial loading time for things that may not even be needed? I think this is exactly the opposite of what I suggested?

[vzybilly]

just adding in my thoughts about a way to ease this, just load all sprites even if a mod doesn't use them, allowing the data step to run more dynamically...

So that would mean increasing the initial loading time for things that may not even be needed? I think this is exactly the opposite of what I suggested?

The initial thing was:

TL;DR
A way to reload data-definitions without restarting Factorio would be very convenient.

in which, the whole issue was that the sprites would need to be updated in the case that a mod was activated/deactivated/now uses the sprite and the sprites take up the whole load time so it was illogical to implement this because it would be the same time as restarting the whole game.

By flat out loading all the images into sprites, the changes to data.raw would be much less meaningful to them, allowing the data scripts to be reloaded right before going into a world or coming out of the mod selection screen. Yes, it will increase the initial load time (by how much?) but it allows re-running the data scripts in afew seconds instead of restarting all of factorio in afew minutes. In my case, I see 3 seconds for data scripts and over 90 seconds for a complete restart of factorio. those three seconds could also be further reduced by knowing what has been changed by what mod (game already does this) and knowing what mods has changed between runs (part to implement). I say, for mod devs atleast, it could be gotten down to almost instant re-runs of the data scripts after the initial load.

[SirRichie]

Oh ok, got it now. That makes a lot of sense then.
Except of course that depending on the amount of disabled mods one might have to spend a lot of time on loading sprites (assuming this assumption is correct).

I think that in such a case, there should really be a background loading process (as suggested in my original post).

[vzybilly]

Oh ok, got it now. That makes a lot of sense then.
Except of course that depending on the amount of disabled mods one might have to spend a lot of time on loading sprites (assuming this assumption is correct).

I think that in such a case, there should really be a background loading process (as suggested in my original post).

it could easily be moved off to another thread and ran in parallel to the startup, just have the main game gui images get loaded up after the scripts and get into the menus and such while the background process works on loading up the mod images from startup. if the user starts a world before the sprites are done loading then show a loading screen for the "mod sprites" with a back button so they could then go fiddle with menus and such if they want while they wait. the only thing the game would really have to do differently is look in all mod locations for any usable image and add it into a table of fileNames -> spriteIndex then when the mods are loaded, they just have the fileName which goes through the table and it gets the sprite that it wants. with this separation, there could be a reload sprites option as well as the reload mods option.

with all this I think there could be a "reload mod" sub-menu somewhere that has individual buttons to reload: data scripts, control scripts, locale, and graphics/sprites.

side note, the control scripts are reloaded each time you enter a world if they are extracted. if they are in a zip then you have to restart the whole game currently...