Page 1 of 1

[MOD 0.14] Data Raw Prototypes

Posted: Sat Dec 10, 2016 9:52 am
by Earendel
Title: Data Raw Prototypes
Short Description: A modding utility to access raw prototype data from data.raw within control.lua. Prototypes are only loaded as required, cached to the save game for as long as the mods state is unchanged, and the cache is shared across all mods. To use: remote.call("data-raw", "prototype", "car", "tank"). See readme for more details.
Name: data-raw-prototypes
Factorio Version: 0.14
Mod State: Beta
Dependencies: Base
Author: Earendel
Downloads: Mod Portal
Licence: See below

Long Description
The mod uses loadstring() to load parts of data.raw that are available to data.lua files. The data is saved in a way that allows loading of specific prototypes instead of needing to load the entire data.raw in one go. This means that if you only need to load a single car type entity, or only need to load a few types of ammo it won’t cause the game to hang for a long time. This mod also uses a cache that lasts as long as prototypes are unchanged, so even if you load the same prototype multiple times (even across different save/load sessions) then the load call will only be performed once. Additionally, this mod provides interfaces for other mods to access the same cache, so even if multiple mods request the same prototype, that prototype will only be loaded once.

Instructions
Access a raw prototype

Code: Select all

remote.call("data-raw", "prototype", "car", "tank")
-- returns the raw prototype from data.raw.car.tank
-- replace "car" with the desired prototype type and "tank" with the desired prototype name. 
Access all prototype of a type

Code: Select all

remote.call("data-raw", "prototypes_of_type", "car")
-- returns the raw prototype dictionary from data.raw.car -- e.g: {car = {car_prototype...}, tank = {tank_prototype...}, ...}
-- replace "car" with the desired prototype type.
Access a list of all prototype types and prototype names

Code: Select all

remote.call("data-raw", "prototypes_list")
-- returns a dictionary of prototype types, each containing a list of prototype names of that type -- e.g: {car = {"car", "tank", ...}, gun = {"shotgun", ...}, ...}
Troubleshooting
The 'game' variable must have loaded in order to access the prototypes, see the http://lua-api.factorio.com/latest/Data-Lifecycle.html for more details.

Avoid using data-final-fixes.lua to alter prototypes, use data-updates.lua instead, otherwise if your mod is processed after this mod then any changes will not be loaded.

Licence

Re: [MOD 0.14] Data Raw Prototypes

Posted: Tue Dec 13, 2016 2:39 am
by Kormer
I'm getting an error from this mod. It's not a big deal since I only have it toggled on when I'm collecting dictionaries for another mod. This is after opening the game and running it immediately. It only happens if this mod is enabled, however I have a large number of mods installed so it could be a combination issue.


320.797 Warning WriteFileGuard.cpp:25: Writing \AppData\Roaming\Factorio\saves\_autosave3.zip failed; previous version (if any) should still be available
320.798 Error ParallelScenarioSaver.cpp:82: Saving scenario failed: Too big amount of parameters in LocalisedString.

Re: [MOD 0.14] Data Raw Prototypes

Posted: Wed Dec 14, 2016 6:39 pm
by Earendel
Kormer wrote:I'm getting an error from this mod. It's not a big deal since I only have it toggled on when I'm collecting dictionaries for another mod. This is after opening the game and running it immediately. It only happens if this mod is enabled, however I have a large number of mods installed so it could be a combination issue.


320.797 Warning WriteFileGuard.cpp:25: Writing \AppData\Roaming\Factorio\saves\_autosave3.zip failed; previous version (if any) should still be available
320.798 Error ParallelScenarioSaver.cpp:82: Saving scenario failed: Too big amount of parameters in LocalisedString.
It looks like one of the other mods has so much data that it can't be saved. Can you include the mod list you're using so I can try and find which ones are causing problems? I may need to manually exclude some prototypes if they are breaking things.

Re: [MOD 0.14] Data Raw Prototypes

Posted: Sun Dec 18, 2016 10:37 pm
by Kormer
I have a ton of mods. It isn't really that big a deal because I'm only loading the mod temporarily to dump stuff out to files for further processing.

Mod List

Re: [MOD 0.14] Data Raw Prototypes

Posted: Mon Dec 19, 2016 3:37 am
by sparr
This is so much fancier than my https://github.com/sparr/factorio-mod-expose-data-raw

Thanks for this!

Re: [MOD 0.14] Data Raw Prototypes

Posted: Mon Dec 19, 2016 11:44 pm
by Earendel
sparr wrote:This is so much fancier than my https://github.com/sparr/factorio-mod-expose-data-raw

Thanks for this!
:D Thanks

Re: [MOD 0.14] Data Raw Prototypes

Posted: Fri May 05, 2017 10:40 am
by EldaRion
Hey Earendel,

Excuse me, I can not write English very well.
I hope "google translate" helps a bit :?

First of all, I would like to praise and praise your mods. THX

I play with some friends on a dedicated server (Factorio 0.15.6 / Linux (Ubuntu) 16.04))
Yesterday we wanted to install all "AAI-Mods". In the singleplayer this works, but in the multiplayer, the mod "Data Raw Prototypes" is a big problem.

I have all Mod's tested separately in the MP. If you enable the "Data Raw Prototypes", the connection to the server ("catch-up") takes very long, about 30 seconds. Without the mod not even 1 second. If the friend is finally connected, it laggt very strong with him. With him, the entire UI can no longer be operated. I play in the local net and can use the UI, but the game is very laggy.

There are also no error messages or errors in the log file.

Unfortunately, the "Data Raw Prototypes" is a prerequisite for "AAI Programmable Vehicles" and "AAI Vehicles: Miner". Especially the miner we would like to use.

I hope you understand what I mean in the bad English and quickly find a solution.

I hope you can fix it. :)

Regards
deutsch

Re: [MOD 0.14] Data Raw Prototypes

Posted: Fri May 05, 2017 4:10 pm
by sparr
Unfortunately this mod adds a LOT of data to a variable that ends up passed between the data and game stages.

The way to fix this problem is for the AAI mod developer to make feature requests to the Factorio devs so that their mod won't need Data Raw Prototypes any more.

Almost any time this mod is necessary, it means someone needs something the game devs haven't implemented yet through the prototypes interface.

Re: [MOD 0.14] Data Raw Prototypes

Posted: Sat May 06, 2017 8:32 am
by Earendel
sparr wrote:Unfortunately this mod adds a LOT of data to a variable that ends up passed between the data and game stages.

The way to fix this problem is for the AAI mod developer to make feature requests to the Factorio devs so that their mod won't need Data Raw Prototypes any more.

Almost any time this mod is necessary, it means someone needs something the game devs haven't implemented yet through the prototypes interface.
Yes that's right, the 0.15 change to reduce the max length of the order string really hurt this mid, causing the number of addition entities required to be multiplied greatly.

Plan Part 1: Help the devs get all of the really important data into the Factorio API so that this mod is not required for AAI.

Plan Part 2: This mod will continue, but it will transition to a state where a dependant mod requests for only specific prototypes and specific properties to be stored ahead of time for use later.

Re: [MOD 0.14] Data Raw Prototypes

Posted: Mon May 08, 2017 10:06 pm
by Earendel
Version 0.3.1 has a major change for other mod makers. Basically data-raw-prototypes only stored data that is specifically requested instead of storing everything. You need to specify both the prototype, and which parameters to sore.

Require prototype properties to be stored

Code: Select all

-- in data.lua
data_raw_require(prototype_type, prototype_name, property_name)
eg:

Code: Select all

data_raw_require("car", "tank", "turn_speed")
-- the turn_speed will be saved from data.raw.car.tank

Code: Select all

data_raw_require("car", {"tank", "car"}, "turn_speed")
-- the turn_speed will be saved from data.raw.car.tank, and data.raw.car.car

Code: Select all

data_raw_require("car", "*", "turn_speed")
-- the turn_speed will be saved from data.raw.car.car, data.raw.car.tank, and any other car-type vehicles added by other mods

Code: Select all

data_raw_require("car", "*", {"turn_speed", "turret_turn_speed"})
-- the turn_speed and turret_turn_speed data will be saved from data.raw.car.car, data.raw.car.tank, and any other car-type vehicles added by other mods
The "*" wildcard means all prototypes of that type. If other mods add new prototypes of that type after you call data_raw_require then the prototype will still be included.

If you change things in data-final-fixes.lua then they will probably not be updated in the stored prototypes, so use data-updates.lua if you can.

If you do need to change things in data-final-fixes.lua then in data.lua set data_raw_compile_last_call to your mod name, then if at the end of data-final-fixes data_raw_compile_last_call is still your mod name then call data_raw_compile() to recompile with all your changes.

Code: Select all

-- data.lua
data_raw_compile_last_call = "my-mod-name"

Code: Select all

-- end of data-final-fixes.lua
if data_raw_compile_last_call == "my-mod-name" then
    data_raw_compile()
end

Re: [MOD 0.14] Data Raw Prototypes

Posted: Mon May 08, 2017 11:19 pm
by TRauMa
Earendel wrote:The "*" wildcard means all prototypes of that type. If other mods add new prototypes of that type after you call data_raw_require then the prototype will still be included.
Do wildcards work for properties, too?

Re: [MOD 0.14] Data Raw Prototypes

Posted: Sun May 14, 2017 6:54 am
by Earendel
TRauMa wrote:
Earendel wrote:The "*" wildcard means all prototypes of that type. If other mods add new prototypes of that type after you call data_raw_require then the prototype will still be included.
Do wildcards work for properties, too?
No, in an attempt to make sure people only include what they absolutely need. If an animation, or other graphic set property gets included, it causes a huge increase in the amount that needs to be stored.

Re: [MOD 0.14] Data Raw Prototypes

Posted: Sat Oct 28, 2017 7:11 pm
by waltv79
We had to remove this mod and all of AAI due to a huge amount of CPU usage. We would love to have AAI back but until the dev of this mod can optimize it to use less CPU our server can't have it. Without the mod CPU is at 74% usage and with Data Raw Prototypes we reach 92% to 95%.

Re: [MOD 0.14] Data Raw Prototypes

Posted: Sat Oct 28, 2017 11:04 pm
by Earendel
What version are you using? Also, to help narrow down the problem have you tried with this mod installed but the rest of AAI removed?