cache data.raw instead of recalculating on every startup

mrudat · Post by **mrudat** » Fri Oct 12, 2018 11:30 pm

I think it would be useful to cache data.raw, given that it can take a significant amount of time to recalculate, especially for more complicated mods.

It could work something like the following

for each mod, checksum all of the .lua files (to account for requiring files)
if the checksums match the cached data.raw, load that from disk and use it to build the in-game prototypes
else, proceed as normal, and cache the data.raw table along with the checksums calculated earlier.

For the benefit of editing a mod that's towards the end of the load order, you could instead do something like the following:

for each step in the normal load process:
- checksum the file before it is executed, and add it to a list of file,checksum pairs
- check for a cached data.raw that matches the file,checksum pairs
  - if one exists, make note of it and continue
  - if one does not exist
    - load the latest data.raw from cache (or start from scratch like normal if there is nothing cached)
    - run the file
    - cache the current state of data.raw
load the latest data.raw from cache if data.raw has not yet been populated (ie. we have everything cached)

It would be well worth compressing the cached data, and there would need to be some expiry mechanism in place.

It would probably only be of interest to heavily modded games, as the base data seems to load fast enough, and it would result in a significant amount of disk space being used.

Post by **Rseding91** » Thu Oct 18, 2018 10:33 pm

I've thought about doing this before but there are a lot more variables that go into making the final result of processing data.raw then just checking if the lua files have changed or not. Mod settings, mod load order, game updates, and mod history all would have to be accounted for adding a lot of complexity to the process when it's really not that slow.

bobingabout · Post by **bobingabout** » Fri Oct 19, 2018 7:58 am

I still think it could be optimised in some way.

For example, instead of requiring a hard game restart, you could just reload and reprocess all the data.raw and other code that loads on game launch, but then instead of having to reload all the imagery, just drop old stuff that isn't needed anymore, and add any new stuff. For me, the graphics loading seems to be the slowest part, possibly because it's loading gigabytes of data into RAM.

eradicator · Post by **eradicator** » Fri Oct 19, 2018 8:20 am

Rseding91 wrote: ↑
Thu Oct 18, 2018 10:33 pm
Mod settings, mod load order, game updates, and mod history all would have to be accounted for adding a lot of complexity to the process when it's really not that slow.

Mod settings is one <50kb file, load order doesn't change as long as no mod versions change, game version is a single variable. Not sure what "mod history" is.

Loading the data stage of the "seablock" modpack takes 20 seconds on my system (and it only has about 30 mods, other people play with 100+ mods), sha256 hashing the whole directory (which includes graphics) takes 2 seconds. So i think there's still potential for a dump "either it's all exactly the same or rebuilt the whole thing" caching mechanism. Intelligent per-file / load order aware caching like the OP describes sounds too complex though.

Are all the versions the same?
Are all the file sizes the same?
Are all the hashes the same?

As soon as one step fails rebuild the cache. Doesn't sound more complex than atlas caching.

Post by **Rseding91** » Fri Oct 19, 2018 10:59 am

eradicator wrote: ↑
Fri Oct 19, 2018 8:20 am

Rseding91 wrote: ↑
Thu Oct 18, 2018 10:33 pm
Mod settings, mod load order, game updates, and mod history all would have to be accounted for adding a lot of complexity to the process when it's really not that slow.
Mod settings is one <50kb file, load order doesn't change as long as no mod versions change, game version is a single variable. Not sure what "mod history" is.

Loading the data stage of the "seablock" modpack takes 20 seconds on my system (and it only has about 30 mods, other people play with 100+ mods), sha256 hashing the whole directory (which includes graphics) takes 2 seconds. So i think there's still potential for a dump "either it's all exactly the same or rebuilt the whole thing" caching mechanism. Intelligent per-file / load order aware caching like the OP describes sounds too complex though.

Are all the versions the same?

Are all the file sizes the same?

Are all the hashes the same?

As soon as one step fails rebuild the cache. Doesn't sound more complex than atlas caching.

Loading the data stage of the "Seablock" mod pack takes 4.991 seconds on my system using 0.16.51. When you say "data stage" are you counting building prototypes, loading sounds, and loading graphics as well?

eradicator · Post by **eradicator** » Fri Oct 19, 2018 11:14 am

Rseding91 wrote: ↑
Fri Oct 19, 2018 10:59 am
Loading the data stage of the "Seablock" mod pack takes 4.991 seconds on my system using 0.16.51. When you say "data stage" are you counting building prototypes, loading sounds, and loading graphics as well?

I measured it again and now i'm at 11seconds, counting from starting the game to the beginning of "loading prototypes", i.e. the point where it suddenly jumps from 3% to 10% and starts "loading sprites". It takes another 11 seconds from there to show the main menu (with enabled atlas caching).

Looking at my folder it might not be the minimal seablock config, here's the mods (mostly bob+angels):

modlist

Edit: 23 seconds when i forget to toggle from "power saving" to "normal" mode (laptop).

Post by **Rseding91** » Fri Oct 19, 2018 11:18 am

You want to look in the log file to measure the actual thing you're looking to measure. From the first line of "Loading mod settings ..." to the last line of "Checksum of ..." is the total time spent building data.raw and the mod change history.

eradicator · Post by **eradicator** » Fri Oct 19, 2018 11:24 am

Rseding91 wrote: ↑
Fri Oct 19, 2018 11:18 am
You want to look in the log file [...]

Was just about to do that. You answer too fast ;)

10.68 seconds from first of "loading mod settings" to last of "checksum". (21.18 seconds in power saving mode).

Core i7-3630QM @2.4Ghz

Factorio Forums

cache data.raw instead of recalculating on every startup

cache data.raw instead of recalculating on every startup

Re: cache data.raw instead of recalculating on every startup

Re: cache data.raw instead of recalculating on every startup

Re: cache data.raw instead of recalculating on every startup

Re: cache data.raw instead of recalculating on every startup

Re: cache data.raw instead of recalculating on every startup

Re: cache data.raw instead of recalculating on every startup

Re: cache data.raw instead of recalculating on every startup