Page 1 of 1
[1.1.92] data.raw JSON dump serializes empty arrays wrong
Posted: Sat Oct 07, 2023 12:55 am
by BrainGamer_
In JSON array types are expected to be something like this
and an empty array should be
but when getting the data.raw JSON dump by using the
--dump-data flag fields that should be arrays and are empty are serialized as
which implies the field to be a map type of sorts and not an array.
Re: [1.1.92] data.raw JSON dump serializes empty arrays wrong
Posted: Sat Oct 07, 2023 3:08 am
by Rseding91
Thanks for the report. Factorio doesn't actually support array types in data.raw. All types in data.raw are dictionaries with anything looking like an array being numeric-dictionaries.
{ value1, value2, ... } is actually short-hand for { 1=value1, 2=value2, 3=... }
Re: [1.1.92] data.raw JSON dump serializes empty arrays wrong
Posted: Sat Oct 07, 2023 7:09 am
by BrainGamer_
Rseding91 wrote: Sat Oct 07, 2023 3:08 am
All types in data.raw are dictionaries with anything looking like an array being numeric-dictionaries
Interesting to know! The prototype docs just specified some types to be
array[union] which is how I ended up at the assumption that they are arrays and that tripped when trying to deserialize "{}" as an array.
Re: [1.1.92] data.raw JSON dump serializes empty arrays wrong
Posted: Sat Oct 07, 2023 2:32 pm
by Rseding91
BrainGamer_ wrote: Sat Oct 07, 2023 7:09 am
Interesting to know! The prototype docs just specified some types to be
array[union] which is how I ended up at the assumption that they are arrays and that tripped when trying to deserialize "{}" as an array.
We use the term "array" to mean "you don't need to care about the key values" but Lua itself only supports dictionaries (actually hash-maps under the hood).
Re: [1.1.92] data.raw JSON dump serializes empty arrays wrong
Posted: Sun Oct 08, 2023 7:33 am
by BrainGamer_
Rseding91 wrote: Sat Oct 07, 2023 2:32 pm
We use the term "array" to mean "you don't need to care about the key values"
Makes sense. I still need to care about it in some way tho since the JSON changes "type" from being an array to a map when the "array" (or internal hash-map) is empty
I found a workaround but it is pretty hacky..
Re: [1.1.92] data.raw JSON dump serializes empty arrays wrong
Posted: Fri Nov 10, 2023 3:54 pm
by moon69
Why is it inconsistent though?
Code: Select all
"flags":
[
"placeable-neutral",
"player-creation"
],
vs
Please can we have empty flags in JSON dump as follows?
Re: [1.1.92] data.raw JSON dump serializes empty arrays wrong
Posted: Fri Nov 10, 2023 4:18 pm
by Rseding91
I went and looked into the "flags" case and found something interesting. Because Lua has no such concept as "dictionary" vs "array" when we parse it from Lua into C++ we have no idea what the type is when it's empty. To Lua, an empty table is an empty table - it's not a dictionary or array. So we have to pick one and when the Lua table is empty it defaults to "empty dictionary" on the C++ side. Along comes the "convert to Json" and it sees an empty dictionary and writes out {}.
To summarize: I don't see this ever changing. Unless Lua gets built in support for JSON (which i'm going to go out on a limb and say it never will).
Re: [1.1.92] data.raw JSON dump serializes empty arrays wrong
Posted: Fri Nov 10, 2023 4:21 pm
by Rseding91
Perhaps a more realistic solution would be writing everything as dictionaries when converting to JSON since that's actually closer to what Lua uses.
Re: [1.1.92] data.raw JSON dump serializes empty arrays wrong
Posted: Fri Nov 10, 2023 4:22 pm
by BrainGamer_
moon69 wrote: Fri Nov 10, 2023 3:54 pm
Code: Select all
"flags":
[
"placeable-neutral",
"player-creation"
],
The fun thing is that this is not the only valid representation for this! Since Lua basically does everything with HashMaps like Rseding said you can have the following:
Code: Select all
"flags":
{
1: "placeable-neutral",
2: "player-creation"
}
or even better:
Code: Select all
"flags":
{
"1": "placeable-neutral",
"2": "player-creation"
}
My solution to all of this in the end was to write a completely custom serde deserializer for this special type:
https://github.com/fgardt/factorio-scan ... #L107-L154
Re: [1.1.92] data.raw JSON dump serializes empty arrays wrong
Posted: Fri Feb 14, 2025 1:23 pm
by Geheim
Ran into this as well with version 2.0.34. Just to understand the dumping process better: Is the JSON generated directly from Lua or with C++ after data.raw was parsed? How are the types from the API docs generated? Is the same type information available during the data dump?
Re: [1.1.92] data.raw JSON dump serializes empty arrays wrong
Posted: Fri Feb 14, 2025 1:49 pm
by Bilka
Geheim wrote: Fri Feb 14, 2025 1:23 pm
Ran into this as well with version 2.0.34. Just to understand the dumping process better: Is the JSON generated directly from Lua or with C++ after data.raw was parsed?
C++ takes the raw data.raw from Lua and dumps it as json. No processing or validation is done, e.g. values that the game only reads as integers will still be Lua's doubles which sometimes leads to confusion:
viewtopic.php?p=593577#p593577.
How are the types from the API docs generated?
The types in the API docs are defined by hand, based on what the C++ source code loads.
Is the same type information available during the data dump?
No. But there are machine readable docs available at
https://lua-api.factorio.com/latest/aux ... otype.html and tools like FMTK use them to support type annotations for Lua:
106458
Re: [1.1.92] data.raw JSON dump serializes empty arrays wrong
Posted: Fri Feb 14, 2025 6:46 pm
by Geheim
Interesting choices, thanks for the quick clarification and references.