[2.0.34] Resolve type discrepancies in the Prototype JSON Format

[Geheim]

When deserializing the data-raw-dump.json file (extracted via the --dump-data command line parameter) with a type safe language, some adjustments have to be made. The following list is complete (for version 2.0.34 without additional mods) and grouped into two main categories, with no specific order otherwise. For each adjustment, one example is given (often including the exact line(s) from the data-raw-dump.json file) with suggestions on how to resolve it, together with other related forum posts if found:

Different types

1. CustomInputPrototype::linked_game_control has an empty string as default value. Remove default or set valid value?
2. CreateTrivialSmokeEffectItem::only_when_visible has a default value of false, but the type is float. This should be a bool instead?
3. One of the timestamps of SingleGraphicLayerProcessionBezierControlPoint has a floating point value on line 912484 ("timestamp": 722.5,). Since MapTick should probably stay an integer, set an integer value for this one case as well?
4. TriggerEffect contains an unknown union member DamageEntityTriggerEffectItem, which seems to be this one without the Entity part. Rename it to DamageEntityTriggerEffectItem? Related: 118305
5. DamageTileTriggerEffectItem has the same type field with value "damage" set, which should be "damage-tile" instead?
6. TechnologySlotStyleSpecification::level_range_offset_y -> line 6958 ("level_range_offset_y": -2.5,). Since this also only appears once, use an integer value for this offset?
7. ItemProductPrototype::amount -> line 76792 ("amount": 1.25,). All three amount values (amount, amount_min, amount_max) should probably be floats? Related: 125229
8. CreateParticleTriggerEffectItem::tail_length_deviation -> line 410653 ("tail_length_deviation": 0.5,). This and tail_length could be floats instead?
9. WorkingVisualisations::shift_animation_waypoint_stop_duration -> line 583922 ("shift_animation_waypoint_stop_duration": 487.5,). Duration usually sounds like a float?
10. BaseAttackParameters::lead_target_for_projectile_delay -> line 941786 ("lead_target_for_projectile_delay": 82.5,). Delay also sounds like a float?
11. TriggerEffectItem::repeat_count -> line 961484 ("repeat_count": 0.65,). There are a lot of other floats for this property, change the type to float? Interestingly, repeat_count_deviation is never a float.
12. Is there a reason why empty arrays are set as {} instead of []? This is kind of against the JSON standard. E.g. the ingredients for the recipe-unknown recipe on line 60888. Use [] for empty arrays instead? Related: 109077

Missing properties/variants

1. Sound is missing a variant for a single filename, e.g. line 28681 ("left_click_sound": "__core__/sound/gui-menu-small.ogg"). Add one similar to SoundDefinition? Related: 117186
2. BoundingBox has a third "unused" orientation, which is still used but not part of the tuple variant, e.g. lines 467071-467083 has an orientation with value 0.125. Would it make sense to add a third variant with {MapPosition, MapPosition, RealOrientation}?
3. ShortcutPrototype::action is missing the variant redo, which is used in line 861459 ("action": "redo",). Add this variant?
4. AchievementPrototypeWithCondition::objective_condition is missing the variant late-research, which is used in line 888343 ("objective_condition": "late-research",). Add this variant?
5. NeighbourConnectableConnectionDefinition::location has a position and direction of type MapPosition, but on line 940107 there is ("direction": 0) and there is no variant that only takes one float. Either allow a single value as variant or set a valid MapPosition?
6. UtilityConstants::huge_animation_sound_area and UtilityConstants::space_platform_default_speed_formula are missing for the default utility-constants from lines 29842-33649. Make them optional?
7. CursorBoxSpecification::rts_selected and CursorBoxSpecification::rts_to_be_selected don't exist in the data-raw-dump.json file, but spidertron_remote_selected and spidertron_remote_to_be_selected do from lines 36373-36610. Rename those fields? Related: 117186
8. EditorControllerPrototype::ignore_surface_conditions is missing for the default editor-controller from lines 43986-44014. Make this optional?
9. SpaceLocationPrototype::gravity_pull is missing for the space-location-unknown space-location from lines 201016-201024. Make this optional? Related: 117186
10. FootstepTriggerEffectItem might contain actions in which case it seems that all inherited properties are not defined, e.g. for the CharacterPrototype::synced_footstep_particle_triggers starting from line 422765. Either don’t inherit from CreateParticleTriggerEffectItem, or make the remaining properties (particle_name and initial_height) also optional?
11. RailPictureSet::rail_endings is missing for the dummy-rail-ramp rail-ramp from lines 681603-681733. Make this optional? Related: 117186
12. AchievementPrototypeWithCondition::objective_condition is missing for the solaris dont-use-entity-in-energy-production-achievement from lines 888008-888023. Make this optional? Related: 117186
13. ProcessionTimeline::audio_events is missing for the timeline of the default-rocket-a procession from lines 908324-908386. Make this optional? Related: 117186
14. SingleGraphicLayerProcessionBezierControlPoint is missing the frame for the podjet_emission single-graphic from lines 908548-908579. Make this optional?

Sorry for the long post, I just wanted to have all findings in one place. Please let me know if you would like to have separate requests and/or more details to (some of) those points. Cheers!

Edit: Added reference for l.

[BrainGamer_]

Hello there fellow strictly typed dump parser

Is there a reason why empty arrays are set as {} instead of []? This is kind of against the JSON standard. E.g. the ingredients for the recipe-unknown recipe on line 60888. Use [] for empty arrays instead?

I ran into this some time ago (109077). The reason for it is quite simple: lua has no concept of arrays. Its tables all the way down, the lua "arrays" are just a special case of tables. Its a bit of a pain to deal with especially since the following is also a valid array representation:

Code: Select all

"flags": {
    1: "placeable-neutral",
    2: "player-creation"
}

With 2.0 getting rid of a lot of value coercions I don't think the second variant I showed in the mentioned thread is still a thing (the numeric keys being strings eg. "1", "2", ...).

In regards to certain properties being documented as integer values but being dumped with decimals: this is also an artifact from lua. There are no integers in lua, all numbers are represented as doubles (at least in the version factorio uses). So the engine does the conversion from float to integer when loading the data.raw into the engine side prototypes (just truncating the decimals).

The other parts do seem like documentation inconsistencies/oversights.
I'd make several separate posts for them tho. Provides better searchability and tracking whether certain things got resolved or not.

What kind of tool/project are you working on that uses data dumps?

[Geheim]

Hello there!
Thanks for the reference and your insights. I was under the assumption that the data dump comes from the engine, but if it comes from Lua then that explains some of the points already. I will follow up on this in your thread.

I'd make several separate posts for them tho. Provides better searchability and tracking whether certain things got resolved or not.

Sounds good, thanks.

What kind of tool/project are you working on that uses data dumps?

I guess you could call it an autopilot for playing the game. Mainly just playing around for now though and it’s more of a proof of concept so far, but as soon as it gets usable in some form I will share it in a separate topic, so stay tuned if that sounds interesting.

[Therenas]

Thanks for the very detailed notes, they are very much appreciated! I went through and crossed out everything that was fixed (potentially only for the next release so you'll have to wait on that to verify.)

It was already clarified that the JSON dump takes values straight from Lua, before it's processed into the game engined, and can thus have differing values to what the engine actually uses. If you want the 'processed' stuff, you'll need to print it yourself via a runtime script, since the runtime API uses the post-engine values obviously. That has other downsides though of course, as not everything is accessible at runtime. Anyways.

Here's some notes on the things that aren't crossed out:

ItemProductPrototype::amount -> line 76792 ("amount": 1.25,). All three amount values (amount, amount_min, amount_max) should probably be floats? Related: 125229

-> They are integers and are parsed as such. This happens because these amounts are auto-generated, although I guess the script that does so could truncate them itself for clarity in this common case. I'll propose that internally.

One of the timestamps of SingleGraphicLayerProcessionBezierControlPoint has a floating point value on line 912484 ("timestamp": 722.5,). Since MapTick should probably stay an integer, set an integer value for this one case as well?

CreateParticleTriggerEffectItem::tail_length_deviation -> line 410653 ("tail_length_deviation": 0.5,). This and tail_length could be floats instead?

WorkingVisualisations::shift_animation_waypoint_stop_duration -> line 583922 ("shift_animation_waypoint_stop_duration": 487.5,). Duration usually sounds like a float?

BaseAttackParameters::lead_target_for_projectile_delay -> line 941786 ("lead_target_for_projectile_delay": 82.5,). Delay also sounds like a float?

TriggerEffectItem::repeat_count -> line 961484 ("repeat_count": 0.65,). There are a lot of other floats for this property, change the type to float? Interestingly, repeat_count_deviation is never a float.

-> All these are integers, it just happens that the game's Lua generates floats sometimes that'll be truncated. Which is fine. Kind of sloppy programming yes, a real issue no.

FootstepTriggerEffectItem might contain actions in which case it seems that all inherited properties are not defined, e.g. for the CharacterPrototype::synced_footstep_particle_triggers starting from line 422765. Either don’t inherit from CreateParticleTriggerEffectItem, or make the remaining properties (particle_name and initial_height) also optional?

-> This is a bit unfortunate, but marking the parent's properties as optional will be more confusing than anything else. The docs format can't deal with these kind of dependencies super well yet, so there's bound to be some edge cases like this.

That should be everything, let me know if there's anything else.