[0.16.26] Desync on using script variable

Place to get help with not working mods / modding interface.
Post Reply
Merssedes
Fast Inserter
Fast Inserter
Posts: 117
Joined: Sun Oct 29, 2017 7:05 pm
Contact:

[0.16.26] Desync on using script variable

Post by Merssedes »

Test scenario:

Code: Select all

 TEST_VALUE = 1 
Sequence of actions:
1) start server using --start-server-load-scenario
2) connect server
3) execute (from client)

Code: Select all

 /c TEST_VALUE = 2 
4) disconnect server
5) connect server
6) execute (from client)

Code: Select all

 /c game.player.print (TEST_VALUE) 
Expected:
* chat message "2"

What I get:
* chat message "1"
* game desyncs
Attachments
desync-report-2018-02-28_22-03-53.zip
(3.17 MiB) Downloaded 108 times

quyxkh
Smart Inserter
Smart Inserter
Posts: 1027
Joined: Sun May 08, 2016 9:01 am
Contact:

Re: [0.16.26] Desync on using script variable

Post by quyxkh »

That's a guaranteed desync, because you're printing a value you've set in the server's environment (with the `/c TEST_VALUE=2`, everything running a map runs in sync, including running every console command) but not reloaded in the newly-connected client. Map globals are the way to share values among all connected environments and have them available to be restored on load. Your scenario's responsible for syncing the client's lua environment with the globally-shared lua environment and never putting any local-only values into the game state, including the console. If you want truly local-only printouts, run your game from a terminal and use the native lua `print` function, not the game engine's game-console print which has to produce identical results in all attached environments.

I currently do this by auto-persisting all my own lua globals not specifically excluded to the map globals in on_init, and reloading them during on_load, there may be a more popular way. Anything that's being set by the scenario script and never changed can be either declared local in the script or (if it needs to be referred to from an event handler) its name added to npg, `game` for instance is provided in handlers but not when control.lua first executes, so it's explicitly marked not-to-be-persisted, this is cut down from real working code so it "should' work, if it breaks I missed something, post if you can't make it work and I'll fix it.:

Code: Select all

local npg={game=true} for k in next,_G do npg[k]=true end  --[[ `_G` is lua-speak for "the current global lua environment table" ]]
--[[ your scenario code, then: ]]
script.on_init(function() for k,v in next,_G do if not npg[k] then global[k]=v] end end)
script.on_load(function() for k,v in next,global do _G[k]=v if global.reload then global.reload() end end)
edit: I haven't changed the above text any, but I snipped my comments-to-myself surrounding the npg assignment above, it's been some weeks and on rereading it seems clear enough to be useful, so here it is:

Code: Select all

-- non-persisted globals table consulted by on_init. All lua globals
-- not added to npg before on_init gets around to it will be persisted
-- in the map's `global` by on_init
local npg = {game=true} for k in next,_G do npg[k]=true end
-- NOTE that lua tables are shallow-copied so saving persists changes
-- in them automatically, but changes to non-table lua globals are not
-- automatically updated in the map globals -- differences between
-- _G.somestring and global.somestring may not be intentional.

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 5206
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: [0.16.26] Desync on using script variable

Post by eradicator »

The real question is...what were you trying to do? If you want players to use commands then add proper commands with LuaCommandProcessor.html and don't use /c.

@quyxkh:
That's certainly worth a place on my personal "most horrible hacks i've seen" list :P.

Merssedes
Fast Inserter
Fast Inserter
Posts: 117
Joined: Sun Oct 29, 2017 7:05 pm
Contact:

Re: [0.16.26] Desync on using script variable

Post by Merssedes »

The source of problem is i tryed to change Oarc settings on the fly.
For example, modify MODULES_ENABLES text to show some info for joining players.

quyxkh
Smart Inserter
Smart Inserter
Posts: 1027
Joined: Sun May 08, 2016 9:01 am
Contact:

Re: [0.16.26] Desync on using script variable

Post by quyxkh »

If I've got the right one, it says in the readme
STEP 2
Go into config.lua and edit the strings to add your own server messages.

Merssedes
Fast Inserter
Fast Inserter
Posts: 117
Joined: Sun Oct 29, 2017 7:05 pm
Contact:

Re: [0.16.26] Desync on using script variable

Post by Merssedes »

quyxkh wrote:If I've got the right one, it says in the readme
STEP 2
Go into config.lua and edit the strings to add your own server messages.
For this i need to restart map, isn't it?

Merssedes
Fast Inserter
Fast Inserter
Posts: 117
Joined: Sun Oct 29, 2017 7:05 pm
Contact:

Re: [0.16.26] Desync on using script variable

Post by Merssedes »

quyxkh wrote:That's a guaranteed desync, because you're printing a value you've set in the server's environment (with the `/c TEST_VALUE=2`, everything running a map runs in sync, including running every console command) but not reloaded in the newly-connected client. Map globals are the way to share values among all connected environments and have them available to be restored on load. Your scenario's responsible for syncing the client's lua environment with the globally-shared lua environment and never putting any local-only values into the game state, including the console. If you want truly local-only printouts, run your game from a terminal and use the native lua `print` function, not the game engine's game-console print which has to produce identical results in all attached environments.
1) Why is it "not reloaded in the newly-connected client"? Isn't savegame (aka downloaded map) contains stript engine state?
2) About "never putting any local-only values into the game state": isn't game state on all clients should be synced by default? I modify scenario variable, which (as I thought) should be updated on all clients accordingly.

quyxkh
Smart Inserter
Smart Inserter
Posts: 1027
Joined: Sun May 08, 2016 9:01 am
Contact:

Re: [0.16.26] Desync on using script variable

Post by quyxkh »

Not everything needs to be synced, control.lua sharply limits what needs explicit syncs to keep network traffic under control. If every variable update in every client was synced it'd go through the roof. OARC looks like it doesn't expect those to change after startup, if you want you can put them all in a table, put the table in global, then updates would be synced. I think the scenario control script itself is part of the map, so you can fix it on your own server copy and run that, clients will get the one you started the map with.

Merssedes
Fast Inserter
Fast Inserter
Posts: 117
Joined: Sun Oct 29, 2017 7:05 pm
Contact:

Re: [0.16.26] Desync on using script variable

Post by Merssedes »

quyxkh wrote:Not everything needs to be synced, control.lua sharply limits what needs explicit syncs to keep network traffic under control. If every variable update in every client was synced it'd go through the roof. OARC looks like it doesn't expect those to change after startup, if you want you can put them all in a table, put the table in global, then updates would be synced. I think the scenario control script itself is part of the map, so you can fix it on your own server copy and run that, clients will get the one you started the map with.
About variable updates, I see it this way. There is 3 sources of changes in non-local variables: game event handlers, console commands (player or server-generated) and player actions.
All game events are deterministic, so their sequence should be the same for all clients and doesn't require external syncing.
Player actions are currently syncing, so they already are part of traffic.
Console command and chat messages, also part of traffic.
So the only time when you need to send current state of script engine (values of all non-local variable) is player connecting to server, i.e. with the map, wich is savegame, wich contains script engine state.
Did i miss something?

Last time i tried to replace control.lua within existing savegame for server (i had problems with uploading savegames to hosting, wich for some reason was replacing control.lua with default one from freeplay mode) I got completely broken game state (event handlers didn't called).

quyxkh
Smart Inserter
Smart Inserter
Posts: 1027
Joined: Sun May 08, 2016 9:01 am
Contact:

Re: [0.16.26] Desync on using script variable

Post by quyxkh »

Merssedes wrote:There is 3 sources of changes in non-local variables: game event handlers, console commands (player or server-generated) and player actions.
All game events are deterministic, so their sequence should be the same for all clients and doesn't require external syncing.
And that looks correct, except the control script has its part to play in keeping the sync overhead down. If it doesn't do it right, it desyncs. Because those variables were never expected to change, they're not sent as part of the sync state, they're just taken as-is from the control script. This cuts down the required traffic. Control scripts can maintain massive local state they've derived from the map, sync-everything is not an option, replay-the-entire-game-history is not an option, there has to be a small, manageable set of values which the control script can combine with the map state itself to produce a nicely-synced local state. That's the map globals, and values that can be changed by user action need to be included in them to avoid every client needing to replay the entire history of user action on connect.

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 5206
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: [0.16.26] Desync on using script variable

Post by eradicator »

When someone joins an MP game the effect is exactly the same as if he was loading a savegame from disk. Except that he loads a savegame that was saved on the host in the moment he pressed the "join" button. This means that anything that does not survive saving the game and reloading normally will also not survive joining MP. Things that survive being saved are: the map data, the maps control.lua ("scenario") and all serializable data in the global lua table called "global".
The internal lua state is never saved or synced to anywhere. It exists only in the running client. And the only data that is transferred between clients during multiplayer are player input actions, as those are the only thing that can not be deterministically predicted.

Merssedes
Fast Inserter
Fast Inserter
Posts: 117
Joined: Sun Oct 29, 2017 7:05 pm
Contact:

Re: [0.16.26] Desync on using script variable

Post by Merssedes »

eradicator wrote:When someone joins an MP game the effect is exactly the same as if he was loading a savegame from disk. Except that he loads a savegame that was saved on the host in the moment he pressed the "join" button. This means that anything that does not survive saving the game and reloading normally will also not survive joining MP. Things that survive being saved are: the map data, the maps control.lua ("scenario") and all serializable data in the global lua table called "global".
The internal lua state is never saved or synced to anywhere. It exists only in the running client. And the only data that is transferred between clients during multiplayer are player input actions, as those are the only thing that can not be deterministically predicted.
So if I assign some handler from console to event, save & reload game, that handler will be gone (just checked it) :shock: For me it's quite unintuitive, but now I know something new and can use it.

Isn't chat & console commands is part of "player input actions"?

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 5206
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: [0.16.26] Desync on using script variable

Post by eradicator »

Merssedes wrote:Isn't chat & console commands is part of "player input actions"?
Yes, chat is an input action. But console commands manipulate the underlying lua state, so they should behave the same as any content of control.lua. That's why's what you did causes a desync. You assign a local variable, (which is not synced as expainled above), and then use that variable in a call to the game state, which is synced (because well, it's the game state). The same thing would happen if you tried to game.print() a local variable in control.lua, except in control.lua it is more difficult to get a local variable to not be the same for everyone that's in the game at the time.

The tl;dr version is... just don't ever use console commands in multiplayer.

Nexela
Smart Inserter
Smart Inserter
Posts: 1828
Joined: Wed May 25, 2016 11:09 am
Contact:

Re: [0.16.26] Desync on using script variable

Post by Nexela »

eradicator wrote:The tl;dr version is... just don't ever use console commands in multiplayer.
Console commands are fine..... Just follow the same logic as any other mod

for variables, if it changes between ticks then it NEEDS to be stored in global.

And Script.on_event and Command.add stuff are not saved so using those from console will cause desyncs

Merssedes
Fast Inserter
Fast Inserter
Posts: 117
Joined: Sun Oct 29, 2017 7:05 pm
Contact:

Re: [0.16.26] Desync on using script variable

Post by Merssedes »

So will I be right in following?
Savefile contains map data, full scenario script and value of global variable.
When savefile is loaded:
1) load global variable
2) run control.lua scenario file as if the game just started (no other variables except global are initialized)
3) call script.on_init handler
4) load map data with racing on_chunk_generated & on_chunk_charted events if required

quyxkh
Smart Inserter
Smart Inserter
Posts: 1027
Joined: Sun May 08, 2016 9:01 am
Contact:

Re: [0.16.26] Desync on using script variable

Post by quyxkh »

You're okay up to step 1 there. See doc-html/Data-Lifecycle.html in your factorio install, or the online version at lua-api.factorio.com/latest. There's a lot of setup that occurs before the map globals are loaded, including running all the control.lua's to set up the bootstrap handlers and initial lua state.

Merssedes
Fast Inserter
Fast Inserter
Posts: 117
Joined: Sun Oct 29, 2017 7:05 pm
Contact:

Re: [0.16.26] Desync on using script variable

Post by Merssedes »

quyxkh wrote:You're okay up to step 1 there. See doc-html/Data-Lifecycle.html in your factorio install, or the online version at lua-api.factorio.com/latest. There's a lot of setup that occurs before the map globals are loaded, including running all the control.lua's to set up the bootstrap handlers and initial lua state.
Thanks for good link, but I didn't quite understand how scenarios fit into that scheme... If I understood it right, then order is following (assuming no mods enabled other then base):
Settings stage and data stage are executed on application load for base & core mods.

On game creation:
1) execute control.lua of scenario
2) call script.on_init handler, defined by scenario (if any)
3) run game

On game loading:
1) migrations for base & code mods applying if required
2) execute control.lua of scenario
3) load global variable for base mod
4) call script.on_load handler, defined by scenario (if any)
5) run game

quyxkh
Smart Inserter
Smart Inserter
Posts: 1027
Joined: Sun May 08, 2016 9:01 am
Contact:

Re: [0.16.26] Desync on using script variable

Post by quyxkh »

I don't know any doc that explicitly deals with scenarios, my impression is the map aka scenario itself is treated as its own mod, and the game engine's "c" command runs in the map's environment. Your sequence looks right to me, but I'm doing this for fun, this is territory where I only fix my own understanding as needed or desired.

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 5206
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: [0.16.26] Desync on using script variable

Post by eradicator »

The scenarios control.lua is treated like any other control.lua from any other mod for loading order purposes. There was a thread about this a while ago, where it was mentioned that scenario control.lua is loaded just before the first mod control.lua. As the lifecycle explains the global table is restored after loading (imagine a require statement) of all control.lua's, but before on_load/on_init. As far as i remember (don't quote me on that) it's just an empty table before that and is overwritten when the actual global is restored.

Post Reply

Return to “Modding help”