[MOD 0.14] Data Raw Prototypes

Topics and discussion about specific mods
User avatar
Earendel
Factorio Staff
Factorio Staff
Posts: 716
Joined: Sun Nov 23, 2014 11:57 am
Contact:

[MOD 0.14] Data Raw Prototypes

Post by Earendel »

Title: Data Raw Prototypes
Short Description: A modding utility to access raw prototype data from data.raw within control.lua. Prototypes are only loaded as required, cached to the save game for as long as the mods state is unchanged, and the cache is shared across all mods. To use: remote.call("data-raw", "prototype", "car", "tank"). See readme for more details.
Name: data-raw-prototypes
Factorio Version: 0.14
Mod State: Beta
Dependencies: Base
Author: Earendel
Downloads: Mod Portal
Licence: See below

Long Description
The mod uses loadstring() to load parts of data.raw that are available to data.lua files. The data is saved in a way that allows loading of specific prototypes instead of needing to load the entire data.raw in one go. This means that if you only need to load a single car type entity, or only need to load a few types of ammo it won’t cause the game to hang for a long time. This mod also uses a cache that lasts as long as prototypes are unchanged, so even if you load the same prototype multiple times (even across different save/load sessions) then the load call will only be performed once. Additionally, this mod provides interfaces for other mods to access the same cache, so even if multiple mods request the same prototype, that prototype will only be loaded once.

Instructions
Access a raw prototype

Code: Select all

remote.call("data-raw", "prototype", "car", "tank")
-- returns the raw prototype from data.raw.car.tank
-- replace "car" with the desired prototype type and "tank" with the desired prototype name. 
Access all prototype of a type

Code: Select all

remote.call("data-raw", "prototypes_of_type", "car")
-- returns the raw prototype dictionary from data.raw.car -- e.g: {car = {car_prototype...}, tank = {tank_prototype...}, ...}
-- replace "car" with the desired prototype type.
Access a list of all prototype types and prototype names

Code: Select all

remote.call("data-raw", "prototypes_list")
-- returns a dictionary of prototype types, each containing a list of prototype names of that type -- e.g: {car = {"car", "tank", ...}, gun = {"shotgun", ...}, ...}
Troubleshooting
The 'game' variable must have loaded in order to access the prototypes, see the http://lua-api.factorio.com/latest/Data-Lifecycle.html for more details.

Avoid using data-final-fixes.lua to alter prototypes, use data-updates.lua instead, otherwise if your mod is processed after this mod then any changes will not be loaded.

Licence
Last edited by Earendel on Sat Dec 17, 2016 11:42 am, edited 1 time in total.
Kormer
Inserter
Inserter
Posts: 30
Joined: Wed Nov 30, 2016 1:34 am
Contact:

Re: [MOD 0.14] Data Raw Prototypes

Post by Kormer »

I'm getting an error from this mod. It's not a big deal since I only have it toggled on when I'm collecting dictionaries for another mod. This is after opening the game and running it immediately. It only happens if this mod is enabled, however I have a large number of mods installed so it could be a combination issue.


320.797 Warning WriteFileGuard.cpp:25: Writing \AppData\Roaming\Factorio\saves\_autosave3.zip failed; previous version (if any) should still be available
320.798 Error ParallelScenarioSaver.cpp:82: Saving scenario failed: Too big amount of parameters in LocalisedString.
User avatar
Earendel
Factorio Staff
Factorio Staff
Posts: 716
Joined: Sun Nov 23, 2014 11:57 am
Contact:

Re: [MOD 0.14] Data Raw Prototypes

Post by Earendel »

Kormer wrote:I'm getting an error from this mod. It's not a big deal since I only have it toggled on when I'm collecting dictionaries for another mod. This is after opening the game and running it immediately. It only happens if this mod is enabled, however I have a large number of mods installed so it could be a combination issue.


320.797 Warning WriteFileGuard.cpp:25: Writing \AppData\Roaming\Factorio\saves\_autosave3.zip failed; previous version (if any) should still be available
320.798 Error ParallelScenarioSaver.cpp:82: Saving scenario failed: Too big amount of parameters in LocalisedString.
It looks like one of the other mods has so much data that it can't be saved. Can you include the mod list you're using so I can try and find which ones are causing problems? I may need to manually exclude some prototypes if they are breaking things.
Kormer
Inserter
Inserter
Posts: 30
Joined: Wed Nov 30, 2016 1:34 am
Contact:

Re: [MOD 0.14] Data Raw Prototypes

Post by Kormer »

I have a ton of mods. It isn't really that big a deal because I'm only loading the mod temporarily to dump stuff out to files for further processing.

Mod List
sparr
Smart Inserter
Smart Inserter
Posts: 1449
Joined: Fri Feb 14, 2014 5:52 pm
Contact:

Re: [MOD 0.14] Data Raw Prototypes

Post by sparr »

This is so much fancier than my https://github.com/sparr/factorio-mod-expose-data-raw

Thanks for this!
User avatar
Earendel
Factorio Staff
Factorio Staff
Posts: 716
Joined: Sun Nov 23, 2014 11:57 am
Contact:

Re: [MOD 0.14] Data Raw Prototypes

Post by Earendel »

sparr wrote:This is so much fancier than my https://github.com/sparr/factorio-mod-expose-data-raw

Thanks for this!
:D Thanks
EldaRion
Burner Inserter
Burner Inserter
Posts: 13
Joined: Fri May 05, 2017 9:31 am
Contact:

Re: [MOD 0.14] Data Raw Prototypes

Post by EldaRion »

Hey Earendel,

Excuse me, I can not write English very well.
I hope "google translate" helps a bit :?

First of all, I would like to praise and praise your mods. THX

I play with some friends on a dedicated server (Factorio 0.15.6 / Linux (Ubuntu) 16.04))
Yesterday we wanted to install all "AAI-Mods". In the singleplayer this works, but in the multiplayer, the mod "Data Raw Prototypes" is a big problem.

I have all Mod's tested separately in the MP. If you enable the "Data Raw Prototypes", the connection to the server ("catch-up") takes very long, about 30 seconds. Without the mod not even 1 second. If the friend is finally connected, it laggt very strong with him. With him, the entire UI can no longer be operated. I play in the local net and can use the UI, but the game is very laggy.

There are also no error messages or errors in the log file.

Unfortunately, the "Data Raw Prototypes" is a prerequisite for "AAI Programmable Vehicles" and "AAI Vehicles: Miner". Especially the miner we would like to use.

I hope you understand what I mean in the bad English and quickly find a solution.

I hope you can fix it. :)

Regards
deutsch
English read and understand yes, do not write so well.
sparr
Smart Inserter
Smart Inserter
Posts: 1449
Joined: Fri Feb 14, 2014 5:52 pm
Contact:

Re: [MOD 0.14] Data Raw Prototypes

Post by sparr »

Unfortunately this mod adds a LOT of data to a variable that ends up passed between the data and game stages.

The way to fix this problem is for the AAI mod developer to make feature requests to the Factorio devs so that their mod won't need Data Raw Prototypes any more.

Almost any time this mod is necessary, it means someone needs something the game devs haven't implemented yet through the prototypes interface.
User avatar
Earendel
Factorio Staff
Factorio Staff
Posts: 716
Joined: Sun Nov 23, 2014 11:57 am
Contact:

Re: [MOD 0.14] Data Raw Prototypes

Post by Earendel »

sparr wrote:Unfortunately this mod adds a LOT of data to a variable that ends up passed between the data and game stages.

The way to fix this problem is for the AAI mod developer to make feature requests to the Factorio devs so that their mod won't need Data Raw Prototypes any more.

Almost any time this mod is necessary, it means someone needs something the game devs haven't implemented yet through the prototypes interface.
Yes that's right, the 0.15 change to reduce the max length of the order string really hurt this mid, causing the number of addition entities required to be multiplied greatly.

Plan Part 1: Help the devs get all of the really important data into the Factorio API so that this mod is not required for AAI.

Plan Part 2: This mod will continue, but it will transition to a state where a dependant mod requests for only specific prototypes and specific properties to be stored ahead of time for use later.
User avatar
Earendel
Factorio Staff
Factorio Staff
Posts: 716
Joined: Sun Nov 23, 2014 11:57 am
Contact:

Re: [MOD 0.14] Data Raw Prototypes

Post by Earendel »

Version 0.3.1 has a major change for other mod makers. Basically data-raw-prototypes only stored data that is specifically requested instead of storing everything. You need to specify both the prototype, and which parameters to sore.

Require prototype properties to be stored

Code: Select all

-- in data.lua
data_raw_require(prototype_type, prototype_name, property_name)
eg:

Code: Select all

data_raw_require("car", "tank", "turn_speed")
-- the turn_speed will be saved from data.raw.car.tank

Code: Select all

data_raw_require("car", {"tank", "car"}, "turn_speed")
-- the turn_speed will be saved from data.raw.car.tank, and data.raw.car.car

Code: Select all

data_raw_require("car", "*", "turn_speed")
-- the turn_speed will be saved from data.raw.car.car, data.raw.car.tank, and any other car-type vehicles added by other mods

Code: Select all

data_raw_require("car", "*", {"turn_speed", "turret_turn_speed"})
-- the turn_speed and turret_turn_speed data will be saved from data.raw.car.car, data.raw.car.tank, and any other car-type vehicles added by other mods
The "*" wildcard means all prototypes of that type. If other mods add new prototypes of that type after you call data_raw_require then the prototype will still be included.

If you change things in data-final-fixes.lua then they will probably not be updated in the stored prototypes, so use data-updates.lua if you can.

If you do need to change things in data-final-fixes.lua then in data.lua set data_raw_compile_last_call to your mod name, then if at the end of data-final-fixes data_raw_compile_last_call is still your mod name then call data_raw_compile() to recompile with all your changes.

Code: Select all

-- data.lua
data_raw_compile_last_call = "my-mod-name"

Code: Select all

-- end of data-final-fixes.lua
if data_raw_compile_last_call == "my-mod-name" then
    data_raw_compile()
end
TRauMa
Inserter
Inserter
Posts: 41
Joined: Fri Jan 15, 2016 3:48 pm
Contact:

Re: [MOD 0.14] Data Raw Prototypes

Post by TRauMa »

Earendel wrote:The "*" wildcard means all prototypes of that type. If other mods add new prototypes of that type after you call data_raw_require then the prototype will still be included.
Do wildcards work for properties, too?
User avatar
Earendel
Factorio Staff
Factorio Staff
Posts: 716
Joined: Sun Nov 23, 2014 11:57 am
Contact:

Re: [MOD 0.14] Data Raw Prototypes

Post by Earendel »

TRauMa wrote:
Earendel wrote:The "*" wildcard means all prototypes of that type. If other mods add new prototypes of that type after you call data_raw_require then the prototype will still be included.
Do wildcards work for properties, too?
No, in an attempt to make sure people only include what they absolutely need. If an animation, or other graphic set property gets included, it causes a huge increase in the amount that needs to be stored.
waltv79
Manual Inserter
Manual Inserter
Posts: 1
Joined: Sat Oct 28, 2017 7:08 pm
Contact:

Re: [MOD 0.14] Data Raw Prototypes

Post by waltv79 »

We had to remove this mod and all of AAI due to a huge amount of CPU usage. We would love to have AAI back but until the dev of this mod can optimize it to use less CPU our server can't have it. Without the mod CPU is at 74% usage and with Data Raw Prototypes we reach 92% to 95%.
User avatar
Earendel
Factorio Staff
Factorio Staff
Posts: 716
Joined: Sun Nov 23, 2014 11:57 am
Contact:

Re: [MOD 0.14] Data Raw Prototypes

Post by Earendel »

What version are you using? Also, to help narrow down the problem have you tried with this mod installed but the rest of AAI removed?
Post Reply

Return to “Mods”