Serialise instances of classes to JSON

[fredthedeadhead]

I want to export as much live data as possible from Factorio. I'm aiming to create a live map of Factorio, in the style of the Minecraft mod Dynmap. I also hope to make a platform that will be a good basis for some other mods, as well as learning some new tech.

To do this I want to react to Factorio events, and convert objects into a common data format (for example, JSON, although some other common format would be good) to be transferred via file or stdout. I'm new to Lua and Factorio modding! I'm hoping there's a good way to do this.

I'm stuck on converting the object to JSON. I've tried serpent.block(obj), but from what I understand it can only serialise tables. LuaPlayer and LuaEntity are not tables - so the result has no useful information.

Code: Select all

-- result of serpent.block(obj)
{
  __self = "userdata"
}

The same is true of game.table_to_json(obj), which returns an error.

Code: Select all

-- result of game.table_to_json(obj)
Value (userdata) can't be saved in property tree at ROOT.__self

I can manually create an LuaPlayer-to-Table mapper function, but creating a mapper function for all classes seems like a lot of work! Before I head down that path I'd to ask if there was some other standard method, or available library, to serialise objects?



Here's a sample of what I have so far. The mod receives an event, and produces a number of files (depending on what the event is referring to).

Code: Select all

-- receive event from Factorio
script.on_event(
    defines.events.on_player_mined_entity,
    function(e)
        entityEvent(e.tick, e.entity, "on_player_mined_entity")
        playerEvent(e.tick, e.player_index, "on_player_mined_entity")
    end
)

-- serialise valid Entity event updates
function entityEvent(tick, entity, eventType)
    if (entity.unit_number ~= nil) and (entity.unit_number ~= 0) then
        local ____ = entity[""]
        local filename = table.concat({entity.unit_number, eventType, tick}, "_")
        outputEventFile((entity.object_name .. "/") .. filename, entity)
    end
end

-- serialise valid player updates, along with the player's character(s)
function playerEvent(tick, player_index, eventType)
    local player = game.players[player_index]
    local filename = table.concat({player.index, eventType, tick}, "_")
    outputEventFile((player.object_name .. "/") .. filename, player)
    if player.character ~= nil then
        entityEvent(tick, player.character, eventType)
        for ____, char in ipairs(
            player.get_associated_characters()
        ) do
            if char ~= nil then
                entityEvent(tick, char, eventType)
            end
        end
    end
end

-- write event to disk (TODO use stdout instead)
function outputEventFile(filename, obj)
    local data = serpent.block(obj) -- doesn't work
    game.write_file((EVENT_FILE_DIR .. "/") .. filename, data, false, 0)
end

[mrvn]

You have to define the fields you want to save for each entity type and loop over them. The objects are some magic glue between lua and the C++ data, they are a black box or oracle that you can ask questions but not save itself.

[fredthedeadhead]

Sorry, I'm confused by your answer. I think you're describing, at a high level, how to convert an object to a table/JSON - that's something I can do manually, but it's a lot of work.

I think it might be possible to generate code programmatically. I'd start by categorising each class into one of three types: 1. Primitives (strings, ints, floats...), 2. Simple data classes (colours, directions...), and 3. Complex classes, with unique IDs (players, entities, surfaces...).

Then, either by parsing help() or the API JSON from the Factorio API site, generate mapper functions for each class that will directly convert primitives and simple data classes into JSON equivalents, and fields with complex classes will only be referenced by ID (hopefully this will avoid circular references). Ideally the mapper class should accept a list of fields to include/exclude.

But yeah, that's a lot of work. The same for manually writing it too. So I wanted to know if there's already a library or method that can convert a object to a Lua-table, or data format (e.g. JSON). I've tried searching but I haven't found anything - only the serpent and table_to_json() methods.

[mrvn]

Primitive and simple data classes you can use serpent.block(). I was talking about the objects with `__self = "userdata"`.

And yes, you can probably parse the API docs to generate code for those automatically. I was just saying you need that code. If you write it by hand or by parsing the API docs is left up to you.

[fredthedeadhead]

I've found this topic which is similar. viewtopic.php?t=74131&p=448975. Rseding91 replied, and said none of the options would happen. I agree that modifying the source classes to add serialisation would be very detrimental, but I'd ask if Factorio could provide a utility class that could serialise classes to Json on request.

In the topic, Opteras-Library is mentioned, and logger.lua certainly gives a good starting point https://github.com/Yousei9/Opteras-Libr ... r.lua#L290

I've also found this library, which I haven't investigated thoroughly but it doesn't look https://github.com/TomCaserta/FactorIO/ ... ialize.lua

I've made some progress in creating serializers manually, for a limited number of fields. I might share it as a library if things work out.

[mrvn]

I've found this topic which is similar. viewtopic.php?t=74131&p=448975. Rseding91 replied, and said none of the options would happen. I agree that modifying the source classes to add serialisation would be very detrimental, but I'd ask if Factorio could provide a utility class that could serialise classes to Json on request.

In the topic, Opteras-Library is mentioned, and logger.lua certainly gives a good starting point https://github.com/Yousei9/Opteras-Libr ... r.lua#L290

I've also found this library, which I haven't investigated thoroughly but it doesn't look https://github.com/TomCaserta/FactorIO/ ... ialize.lua

I've made some progress in creating serializers manually, for a limited number of fields. I might share it as a library if things work out.

Not sure what you are doing. But have you tried simply creating a list of fields you want to serialize for e.g. an inserter. Then loop over that list and for each field store serpent.block(entity[field])?

[fredthedeadhead]

Not sure what you are doing. But have you tried simply creating a list of fields you want to serialize for e.g. an inserter. Then loop over that list and for each field store serpent.block(entity[field])?

I haven't tried that, I'm new to Lua so would you mind giving an example?

I've managed to get something working using TypeScriptToLua. Both these TypeScript functions convert a Lua object to a TS object, which can be converted with table_to_json.

Code: Select all

    export function playerToTable(player: LuaPlayer): JsonTable {
      let charIds = Entity.entitiesToUnitNumbers(player.get_associated_characters())
      return {
        "name": player.name,
        "character_unit_number": player.character?.unit_number,
        "associated_characters_unit_numbers": charIds,
        "position": DataClasses.positionTableToTable(player.position),
      }
    }

    export function entityToTable(entity: LuaEntity): JsonTable {

      let player: JsonTable =
          entity.is_player() ? {
            "player_index": entity.player?.index,
          } : {}

      return {
        player,
        "name": entity.name,
        "type": entity.type,
        "active": entity.active,
        "health": entity.health,
        "surface_index": entity.surface.index,
        "unit_number": entity.unit_number,
        "position": DataClasses.positionTableToTable(entity.position),
      }
    }

The Json gets printed using localised_print. Here's what the output looks like. It's pretty neat! The next step was sending the logs to a Kafka topic, and that was super easy, so as far as a POC goes I'm making good progress.

Code: Select all

{
  "tick": 1290393,
  "object_name": "LuaPlayer",
  "eventType": "on_player_changed_position",
  "data": {
    "name": "fredthedeadhead",
    "characterId": 13,
    "associatedCharacterIds": {},
    "position": {
      "x": 23.625,
      "y": -13.08203125
    }
  }
}
{
  "tick": 1290393,
  "object_name": "LuaEntity",
  "eventType": "on_player_changed_position",
  "data": {
    "player": {},
    "name": "character",
    "type": "character",
    "active": true,
    "health": 250,
    "surfaceId": 1,
    "unit_number": 13,
    "position": {
      "x": 23.625,
      "y": -13.08203125
    }
  }
}
{
  "tick": 1290420,
  "object_name": "LuaSurface",
  "eventType": "on_tick",
  "data": {
    "name": "nauvis",
    "index": 1,
    "daytime": 0.31672000005162126
  }
}

[mrvn]

I'm afraid if you don't know enough LUA to make a list and loop over it then it's better to find some LUA tutorials and learn it properly. I'm nt giving you fish but telling you to buy a fishing book.

[fredthedeadhead]

In case this is useful for others, I've found two other serialisation files in Clusterio

clusterio/packages/slave/modules/clusterio/serialize.lua


clusterio/plugins/inventory_sync/module/serialize.lua