[1.1.92] data.raw JSON dump serializes empty arrays wrong

[BrainGamer_]

In JSON array types are expected to be something like this

Code: Select all

[ value1, value2, ... ]

and an empty array should be

Code: Select all

[]

but when getting the data.raw JSON dump by using the --dump-data flag fields that should be arrays and are empty are serialized as

Code: Select all

{}

which implies the field to be a map type of sorts and not an array.

[Rseding91]

Thanks for the report. Factorio doesn't actually support array types in data.raw. All types in data.raw are dictionaries with anything looking like an array being numeric-dictionaries.

{ value1, value2, ... } is actually short-hand for { 1=value1, 2=value2, 3=... }

[BrainGamer_]

All types in data.raw are dictionaries with anything looking like an array being numeric-dictionaries

Interesting to know! The prototype docs just specified some types to be array[union] which is how I ended up at the assumption that they are arrays and that tripped when trying to deserialize "{}" as an array.

[Rseding91]

Interesting to know! The prototype docs just specified some types to be array[union] which is how I ended up at the assumption that they are arrays and that tripped when trying to deserialize "{}" as an array.

We use the term "array" to mean "you don't need to care about the key values" but Lua itself only supports dictionaries (actually hash-maps under the hood).

[BrainGamer_]

We use the term "array" to mean "you don't need to care about the key values"

Makes sense. I still need to care about it in some way tho since the JSON changes "type" from being an array to a map when the "array" (or internal hash-map) is empty

I found a workaround but it is pretty hacky..

[moon69]

Why is it inconsistent though?

Code: Select all

      "flags": 
      [
        "placeable-neutral",
        "player-creation"
      ],

vs

Code: Select all

      "flags": 
      {},

Please can we have empty flags in JSON dump as follows?

Code: Select all

 "flags": 
      [],

[Rseding91]

I went and looked into the "flags" case and found something interesting. Because Lua has no such concept as "dictionary" vs "array" when we parse it from Lua into C++ we have no idea what the type is when it's empty. To Lua, an empty table is an empty table - it's not a dictionary or array. So we have to pick one and when the Lua table is empty it defaults to "empty dictionary" on the C++ side. Along comes the "convert to Json" and it sees an empty dictionary and writes out {}.

To summarize: I don't see this ever changing. Unless Lua gets built in support for JSON (which i'm going to go out on a limb and say it never will).

[Rseding91]

Perhaps a more realistic solution would be writing everything as dictionaries when converting to JSON since that's actually closer to what Lua uses.

[BrainGamer_]

Code: Select all

"flags": 
      [
        "placeable-neutral",
        "player-creation"
      ],

The fun thing is that this is not the only valid representation for this! Since Lua basically does everything with HashMaps like Rseding said you can have the following:

Code: Select all

"flags": 
      {
        1: "placeable-neutral",
        2: "player-creation"
      }

or even better:

Code: Select all

"flags": 
      {
        "1": "placeable-neutral",
        "2": "player-creation"
      }

My solution to all of this in the end was to write a completely custom serde deserializer for this special type: https://github.com/fgardt/factorio-scan ... #L107-L154

[Geheim]

Ran into this as well with version 2.0.34. Just to understand the dumping process better: Is the JSON generated directly from Lua or with C++ after data.raw was parsed? How are the types from the API docs generated? Is the same type information available during the data dump?

[Bilka]

Ran into this as well with version 2.0.34. Just to understand the dumping process better: Is the JSON generated directly from Lua or with C++ after data.raw was parsed?

C++ takes the raw data.raw from Lua and dumps it as json. No processing or validation is done, e.g. values that the game only reads as integers will still be Lua's doubles which sometimes leads to confusion: viewtopic.php?p=593577#p593577.

How are the types from the API docs generated?

The types in the API docs are defined by hand, based on what the C++ source code loads.

Is the same type information available during the data dump?

No. But there are machine readable docs available at https://lua-api.factorio.com/latest/aux ... otype.html and tools like FMTK use them to support type annotations for Lua: 106458

[Geheim]

Interesting choices, thanks for the quick clarification and references.