Understand data.raw entity pictures

[CheeseMcBurger]

I'm trying to programmatically find the entity pictures from the data.raw table.

If I take the "big-electric-pole" table for example, it has this property:

Code: Select all

      pictures = {
        layers = {
          {
            filename = "__base__/graphics/entity/big-electric-pole/big-electric-pole.png",
            -- shortened for brevity
          },
          {
            draw_as_shadow = true,
            filename = "__base__/graphics/entity/big-electric-pole/big-electric-pole-shadow.png",
            -- shortened for brevity
          }
        }
      }

The "fast-inserter" uses a different table layout that uses a table named "platform_picture", but also a lot of other tables with various different prefixes "*_picture".

Code: Select all

      platform_picture = {
        sheet = {
          filename = "__base__/graphics/entity/fast-inserter/fast-inserter-platform.png",
          -- shortened for brevity
        }
        -- shortened for brevity
      }

Is there any logic approach I can follow to get the sprites programmatically without having to hand-craft all my functions? (I'm trying to create a basic blueprint renderer that gets its data programmatically from the data.raw as a side-project).

[Deadlock989]

Is there any logic approach I can follow to get the sprites programmatically without having to hand-craft all my functions? (I'm trying to create a basic blueprint renderer that gets its data programmatically from the data.raw as a side-project).

No. While some entities do have the same specification as some other entities, there are many entities which deviate to a greater or lesser extent, and a few which have their own unique properties that don't apply to any other entity. For some entities there are additionally legacy properties which provide an alternative method so you would have to handle those as well if you really want to handle all cases, but they are rarely used. Your best bet is to identify common properties shared by multiple entities (e.g. based on the property type such as Animation, RotatedSprite etc. etc.) and then handle every entity type individually, calling your common functions where appropriate. Many entities use animations rather than pictures and some have additional "working visualisation" animations which are only active when the machine is running, but those can also specify an "always on" picture/animation which is drawn regardless.

[CheeseMcBurger]

> For some entities there are additionally legacy properties which provide an alternative method so you would have to handle those as well if you really want to handle all cases, but they are rarely used.

Isn't there a common method that's used for blueprints? There's no animations in blueprints for example and a default non-animated graphic is used.

[DaveMcW]

Isn't there a common method that's used for blueprints? There's no animations in blueprints for example and a default non-animated graphic is used.

Blueprints don't have a complete renderer, they use the surface renderer with minor changes.

The C++ surface renderer uses the same process Deadlock described. Unique graphics code for every entity type, with shared functions where there is overlap.

[BrainGamer_]

Isn't there a common method that's used for blueprints? There's no animations in blueprints for example and a default non-animated graphic is used.

as DaveMcW mentioned it is dependent on the entities type. If you want to have a look at how the various external BP rendering tools do it you can read the java code of it here.
I'm currently in the process of writing my own renderer in rust that should properly support modded entities aswell and will publish that on github at somepoint too.

[CheeseMcBurger]

BrainGamer_, how far are you with that? I'm no Rust developer and just used my side project as a means to learn some Rust and also create a renderer that does certain things I need. Maybe we could team up, although I admit that I'm probably of little help.

But I already have a small LUA script that is able to create the data.raw table by pointing it at the data.lua of a mod. My next step was to implement said script with the mlua crate. Would you care to share what you have and I can see if there's anything I can contribute to.

Edit: Also thanks to all replies and explanations on how the picture stuff works!

[BrainGamer_]

ow far are you with that?

I'm at the point where I am starting to write the rendering stuff of invidivual entities / graphics types. I pretty much completed the data dump deserializer, locale dump deserializer, blueprint string deserializer and sprite loader (from base/core or even from zipped mods).

Maybe we could team up

I'm not sure how useful that could be at this time. There is still tons to do but organizing that with someone else remotely to figure out what should be done next etc etc seems more overhead than it would be worth at the moment. I have not fully figured the system I'll end up with myself.

Would you care to share what you have and I can see if there's anything I can contribute to.

I will certainly put it up on GitHub. Maybe I'll use tomorrow to do some cleanup to get it ready for that sooner since my current plan was to only make the repo public once I am pretty much done with most things.

My next step was to implement said script with the mlua crate

Factorio uses its own Lua variant which is public here. Someone already made a mlua fork that uses factorios special lua here.

[CheeseMcBurger]

I guess I just wait then

[BrainGamer_]

I guess I just wait then

Sorry it took so long but I finally got around to publish the repository. You can find it here: https://github.com/fgardt/factorio-scanner.

It can currently render pretty much all entities apart from rails, vehicles, and wires. I also still need to find a way to handle things like loaders and splitters with 2 belts below them to render them properly. Rendering of connected variants is also not handled yet but only needs the logic to scan the BP to figure out which directions are connected, the rest with choosing the correct variant is done already.

Please let me know any feedback you might have!

[CheeseMcBurger]

Do I correctly understand that you expect the prototype instance data of a given mod to be in the file `../dumps/data-raw-dump.json`? And that creating that file from your app is still a to do?

I have code ready that can pull the data from any mod without starting the game. Would that be interesting to you?

[BrainGamer_]

Do I correctly understand that you expect the prototype instance data of a given mod to be in the file `../dumps/data-raw-dump.json`?

I currently have a prototype dump json in the given location but that is not added in the repo.

And that creating that file from your app is still a to do?

Yes this is still not implemented at all. It will involve downloading all mods & their dependencies, update the mod-list.json and mod-settings.dat files and then launch factorio with the prototype dump flag. Later on I plan to have some predefined dumps for common modpacks that you can just select similar to how factoriolab does it.

I have code ready that can pull the data from any mod without starting the game. Would that be interesting to you?

That seems certainly interesting yeah. Is it similar to https://github.com/JohnTheCoolingFan/factorio-lib-rs by running the mods settings / data stage lua files and extracting the data table from that?

In the end I'll still need a factorio install since I'll need all the sprites for vanilla entities/items/etc and by using the games own prototype dumper I can also get some data stage validation aswell.

[CheeseMcBurger]

That seems certainly interesting yeah. Is it similar to https://github.com/JohnTheCoolingFan/factorio-lib-rs by running the mods settings / data stage lua files and extracting the data table from that?

In the end I'll still need a factorio install since I'll need all the sprites for vanilla entities/items/etc and by using the games own prototype dumper I can also get some data stage validation aswell.

I have not actually done it in Rust yet. I have LUA code that I wanted to port to Rust using the mlua crate, but when I heard you're already so much farther when I was only at the beginning, I stopped.

This is the LUA code that practically loads all the data into the `bp_entites` variable. This can be executed using mlua and then you can work with the data in Rust directly. If you need to read mod data, the code can be extended to scan through any mod directory as well.

Code: Select all

-- GAME_FOLDER is the only thing that needs to be set for this script to work
GAME_FOLDER = "C:\\Program Files (x86)\\GOG Galaxy\\Games\\Factorio\\"

package.path = GAME_FOLDER .. "data\\base\\?.lua;" .. package.path
package.path = GAME_FOLDER .. "data\\core\\lualib\\?.lua;" .. package.path


-- Insert a custom loader function as the first searcher
table.insert(package.searchers, 2,
    function (modulename)
        local base_path = GAME_FOLDER .. "data\\base"
        local filename = base_path .. modulename:gsub("__base__", "") .. ".lua"
        local file = io.open(filename, "rb")
        if file then
            -- Compile and return the module …
            return assert(loadstring(assert(file:read("*a")), filename))
        else
            -- or return an error message.
            return "\n\tno file '"..filename.."' (checked with factorio-compatible loader)"
        end
    end
)

local function nestable(t)
    return setmetatable(t or {}, {
        __index = function (self, key)
            local new = nestable {}
            rawset(self, key, new)
            return new
        end
    })
end

defines = nestable {}
require("dataloader")
data.raw = nestable {}
require("data")

bp_entities = {} -- contains the entire data

-- Iterate through entity groups
for k,entity_group in pairs(data.raw) do
    -- Iterate through entities
    for k,entity in pairs(entity_group) do
        -- 
        if entity["flags"] ~= nil then

            found_player_creation = nil
            found_not_blueprintable = nil

            for k,flag in pairs(entity.flags) do
                if flag == "player-creation" then
                    found_player_creation = true
                end
                if flag == "not-blueprintable" then
                    found_not_blueprintable = true
                end
                if found_player_creation and not found_not_blueprintable then
                    bp_entities[entity.name] = entity
                end
            end            
        end        
    end
end

But reading your reply again, I realise now there is a command line option `--dump-data` that does a similar thing. I'm not sure what's the better approach.

Starting Factorio with --dump-data requires editing the mod-settings.dat, then starting a Factorio process, waiting for it to finish to digest the output json file.

My approach may be less reliable, but doesn't requires editing the mod-settings.dat and running a Factorio process.

I'll just leave it here ...