Serialise instances of classes to JSON

Place to get help with not working mods / modding interface.
Post Reply
fredthedeadhead
Inserter
Inserter
Posts: 22
Joined: Mon Oct 18, 2021 6:13 pm
Contact:

Serialise instances of classes to JSON

Post by fredthedeadhead »

I want to export as much live data as possible from Factorio. I'm aiming to create a live map of Factorio, in the style of the Minecraft mod Dynmap. I also hope to make a platform that will be a good basis for some other mods, as well as learning some new tech.

To do this I want to react to Factorio events, and convert objects into a common data format (for example, JSON, although some other common format would be good) to be transferred via file or stdout. I'm new to Lua and Factorio modding! I'm hoping there's a good way to do this.

I'm stuck on converting the object to JSON. I've tried serpent.block(obj), but from what I understand it can only serialise tables. LuaPlayer and LuaEntity are not tables - so the result has no useful information.

Code: Select all

-- result of serpent.block(obj)
{
  __self = "userdata"
}
The same is true of game.table_to_json(obj), which returns an error.

Code: Select all

-- result of game.table_to_json(obj)
Value (userdata) can't be saved in property tree at ROOT.__self
I can manually create an LuaPlayer-to-Table mapper function, but creating a mapper function for all classes seems like a lot of work! Before I head down that path I'd to ask if there was some other standard method, or available library, to serialise objects?



Here's a sample of what I have so far. The mod receives an event, and produces a number of files (depending on what the event is referring to).

Code: Select all


-- receive event from Factorio
script.on_event(
    defines.events.on_player_mined_entity,
    function(e)
        entityEvent(e.tick, e.entity, "on_player_mined_entity")
        playerEvent(e.tick, e.player_index, "on_player_mined_entity")
    end
)

-- serialise valid Entity event updates
function entityEvent(tick, entity, eventType)
    if (entity.unit_number ~= nil) and (entity.unit_number ~= 0) then
        local ____ = entity[""]
        local filename = table.concat({entity.unit_number, eventType, tick}, "_")
        outputEventFile((entity.object_name .. "/") .. filename, entity)
    end
end

-- serialise valid player updates, along with the player's character(s)
function playerEvent(tick, player_index, eventType)
    local player = game.players[player_index]
    local filename = table.concat({player.index, eventType, tick}, "_")
    outputEventFile((player.object_name .. "/") .. filename, player)
    if player.character ~= nil then
        entityEvent(tick, player.character, eventType)
        for ____, char in ipairs(
            player.get_associated_characters()
        ) do
            if char ~= nil then
                entityEvent(tick, char, eventType)
            end
        end
    end
end

-- write event to disk (TODO use stdout instead)
function outputEventFile(filename, obj)
    local data = serpent.block(obj) -- doesn't work
    game.write_file((EVENT_FILE_DIR .. "/") .. filename, data, false, 0)
end


mrvn
Smart Inserter
Smart Inserter
Posts: 5112
Joined: Mon Sep 05, 2016 9:10 am
Contact:

Re: Serialise instances of classes to JSON

Post by mrvn »

You have to define the fields you want to save for each entity type and loop over them. The objects are some magic glue between lua and the C++ data, they are a black box or oracle that you can ask questions but not save itself.

fredthedeadhead
Inserter
Inserter
Posts: 22
Joined: Mon Oct 18, 2021 6:13 pm
Contact:

Re: Serialise instances of classes to JSON

Post by fredthedeadhead »

Sorry, I'm confused by your answer. I think you're describing, at a high level, how to convert an object to a table/JSON - that's something I can do manually, but it's a lot of work.

I think it might be possible to generate code programmatically. I'd start by categorising each class into one of three types: 1. Primitives (strings, ints, floats...), 2. Simple data classes (colours, directions...), and 3. Complex classes, with unique IDs (players, entities, surfaces...).

Then, either by parsing help() or the API JSON from the Factorio API site, generate mapper functions for each class that will directly convert primitives and simple data classes into JSON equivalents, and fields with complex classes will only be referenced by ID (hopefully this will avoid circular references). Ideally the mapper class should accept a list of fields to include/exclude.

But yeah, that's a lot of work. The same for manually writing it too. So I wanted to know if there's already a library or method that can convert a object to a Lua-table, or data format (e.g. JSON). I've tried searching but I haven't found anything - only the serpent and table_to_json() methods.

mrvn
Smart Inserter
Smart Inserter
Posts: 5112
Joined: Mon Sep 05, 2016 9:10 am
Contact:

Re: Serialise instances of classes to JSON

Post by mrvn »

Primitive and simple data classes you can use serpent.block(). I was talking about the objects with `__self = "userdata"`.

And yes, you can probably parse the API docs to generate code for those automatically. I was just saying you need that code. If you write it by hand or by parsing the API docs is left up to you.

fredthedeadhead
Inserter
Inserter
Posts: 22
Joined: Mon Oct 18, 2021 6:13 pm
Contact:

Re: Serialise instances of classes to JSON

Post by fredthedeadhead »

I've found this topic which is similar. viewtopic.php?t=74131&p=448975. Rseding91 replied, and said none of the options would happen. I agree that modifying the source classes to add serialisation would be very detrimental, but I'd ask if Factorio could provide a utility class that could serialise classes to Json on request.

In the topic, Opteras-Library is mentioned, and logger.lua certainly gives a good starting point https://github.com/Yousei9/Opteras-Libr ... r.lua#L290

I've also found this library, which I haven't investigated thoroughly but it doesn't look https://github.com/TomCaserta/FactorIO/ ... ialize.lua

I've made some progress in creating serializers manually, for a limited number of fields. I might share it as a library if things work out.

mrvn
Smart Inserter
Smart Inserter
Posts: 5112
Joined: Mon Sep 05, 2016 9:10 am
Contact:

Re: Serialise instances of classes to JSON

Post by mrvn »

fredthedeadhead wrote:
Wed Oct 20, 2021 6:57 pm
I've found this topic which is similar. viewtopic.php?t=74131&p=448975. Rseding91 replied, and said none of the options would happen. I agree that modifying the source classes to add serialisation would be very detrimental, but I'd ask if Factorio could provide a utility class that could serialise classes to Json on request.

In the topic, Opteras-Library is mentioned, and logger.lua certainly gives a good starting point https://github.com/Yousei9/Opteras-Libr ... r.lua#L290

I've also found this library, which I haven't investigated thoroughly but it doesn't look https://github.com/TomCaserta/FactorIO/ ... ialize.lua

I've made some progress in creating serializers manually, for a limited number of fields. I might share it as a library if things work out.
Not sure what you are doing. But have you tried simply creating a list of fields you want to serialize for e.g. an inserter. Then loop over that list and for each field store serpent.block(entity[field])?

fredthedeadhead
Inserter
Inserter
Posts: 22
Joined: Mon Oct 18, 2021 6:13 pm
Contact:

Re: Serialise instances of classes to JSON

Post by fredthedeadhead »

mrvn wrote:
Wed Oct 20, 2021 9:20 pm
Not sure what you are doing. But have you tried simply creating a list of fields you want to serialize for e.g. an inserter. Then loop over that list and for each field store serpent.block(entity[field])?
I haven't tried that, I'm new to Lua so would you mind giving an example?

I've managed to get something working using TypeScriptToLua. Both these TypeScript functions convert a Lua object to a TS object, which can be converted with table_to_json.

Code: Select all

    export function playerToTable(player: LuaPlayer): JsonTable {
      let charIds = Entity.entitiesToUnitNumbers(player.get_associated_characters())
      return {
        "name": player.name,
        "character_unit_number": player.character?.unit_number,
        "associated_characters_unit_numbers": charIds,
        "position": DataClasses.positionTableToTable(player.position),
      }
    }

    export function entityToTable(entity: LuaEntity): JsonTable {

      let player: JsonTable =
          entity.is_player() ? {
            "player_index": entity.player?.index,
          } : {}

      return {
        player,
        "name": entity.name,
        "type": entity.type,
        "active": entity.active,
        "health": entity.health,
        "surface_index": entity.surface.index,
        "unit_number": entity.unit_number,
        "position": DataClasses.positionTableToTable(entity.position),
      }
    }

The Json gets printed using localised_print. Here's what the output looks like. It's pretty neat! The next step was sending the logs to a Kafka topic, and that was super easy, so as far as a POC goes I'm making good progress.

Code: Select all

{
  "tick": 1290393,
  "object_name": "LuaPlayer",
  "eventType": "on_player_changed_position",
  "data": {
    "name": "fredthedeadhead",
    "characterId": 13,
    "associatedCharacterIds": {},
    "position": {
      "x": 23.625,
      "y": -13.08203125
    }
  }
}
{
  "tick": 1290393,
  "object_name": "LuaEntity",
  "eventType": "on_player_changed_position",
  "data": {
    "player": {},
    "name": "character",
    "type": "character",
    "active": true,
    "health": 250,
    "surfaceId": 1,
    "unit_number": 13,
    "position": {
      "x": 23.625,
      "y": -13.08203125
    }
  }
}
{
  "tick": 1290420,
  "object_name": "LuaSurface",
  "eventType": "on_tick",
  "data": {
    "name": "nauvis",
    "index": 1,
    "daytime": 0.31672000005162126
  }
}
Last edited by fredthedeadhead on Fri Oct 22, 2021 6:09 am, edited 1 time in total.

mrvn
Smart Inserter
Smart Inserter
Posts: 5112
Joined: Mon Sep 05, 2016 9:10 am
Contact:

Re: Serialise instances of classes to JSON

Post by mrvn »

I'm afraid if you don't know enough LUA to make a list and loop over it then it's better to find some LUA tutorials and learn it properly. I'm nt giving you fish but telling you to buy a fishing book. :)

fredthedeadhead
Inserter
Inserter
Posts: 22
Joined: Mon Oct 18, 2021 6:13 pm
Contact:

Re: Serialise instances of classes to JSON

Post by fredthedeadhead »

In case this is useful for others, I've found two other serialisation files in Clusterio

clusterio/packages/slave/modules/clusterio/serialize.lua


clusterio/plugins/inventory_sync/module/serialize.lua

Post Reply

Return to “Modding help”