[1.1.33] Rcon polling for data does not "respect" pause at all

[KeepResearchinSpoons]

the funniest demo of these all is this one.
So very much TLDR!
The forum engine does not allow me to embed the ah wait, let's use the weird self-reference for the file we've just loaded...
here:

Boy: look, mommy! I have a game paused!
Mommy: But it still does run as usual?

rcon_unpause.webm

(the same one, I've just embedded it)

(367.73 KiB) Downloaded 145 times

lol


Shall paste details in the next message.
> The history of mishaps... No, spare me this one, it is a huge fine print.
> What tests I've done already
> What it affects all the time around
And even
> what a dirty solution may possibly save half your day, if it also affects your setup with rcon stats you collect (spoiler: it does)

[KeepResearchinSpoons]

It all started with servers magically running even when noone was around.
The is_the_server_alive poll in this story happens once a sec.
(In the video above it happens ~120 times a sec, so you can see it in the real time.)

The fixes for events that failed when there was not a player were introduced,
the problem of the servers still running, bitters evolving, miners depleting etc remained though.

In the nutshell, being polled once per sec made them run at 1/60 the real speed.
So in an hour a minute passed by.

Over the night with 8 hours it was a 10 minutes.
Not exactly enough to kill a well-defended base.
Well, if the scenario was a hard one, with a constant pressure...
All hail the ruins, welcome to the next run! (The fish market is died, soft-reset initiated!)
Or welcome to the half-dead market that may still be possible to save. if you were lucky that is.

And the bug was well known but not confirmed till this week.

This is what exactly an auto-pause exists for, you prevent the map from running when no one is around.
As you can see in the vid, it is not the option.
* provided, you want to know admin-side, if your server is alive.
* or you want to update the server banlist via lua (via rcon)
* or if you update "Trusted" players list globally over all the servers;
* or if you keep the "ups" being measured (this is even more buggy due to the issue)

[KeepResearchinSpoons]

The current workaround

You can't prevent the game time from rising. Sorry.
But! You can use the "editor" option to prevent the ticks from affecting entities!

lua-api for game.tick_paused yay! It is RW!
It is dirty, since it uses some hacks, but here's the idea:

hana_rcon_fix.png (11.66 KiB) Viewed 2832 times

This prevents the game.tick from running on each rcon poll.
Should also prevent on_tick, but well you can add `if game.tick < next_tick` or something for the worst case anyways.
Same way as `/editor` works actually.

You still have the `game.ticks_played` increase with each rcon request
So do the server-time shown in the lobby (and `/time`) increase.
At least you can save yourself the trouble of all soft-resets occured for the bitters...

[KeepResearchinSpoons]

Now for the tests having been made.
If you poll faster than 60/sec,
both game.tick and game.ticks_played increase no faster than the current game.speed.

You can't set game.speed lower than 0.01, so this was not an option for the "real_pause module".

However when you do set paused to true ( or effectively use an /editor, same )
you get

Code: Select all

[    10]@2021-05-10<UTC>19:32 >> {"r_id":13,"ct":2559,"ctp":2707,"t2r":0,"istp":true}
[    11]@2021-05-10<UTC>19:32 >> {"r_id":18,"ct":2559,"ctp":2708,"t2r":0,"istp":true}
[    12]@2021-05-10<UTC>19:32 >> {"r_id":19,"ct":2559,"ctp":2708,"t2r":0,"istp":true}
[    13]@2021-05-10<UTC>19:32 >> {"r_id":20,"ct":2559,"ctp":2708,"t2r":0,"istp":true}
[    14]@2021-05-10<UTC>19:32 >> {"r_id":22,"ct":2559,"ctp":2709,"t2r":0,"istp":true}
[    15]@2021-05-10<UTC>19:32 >> {"r_id":23,"ct":2559,"ctp":2709,"t2r":0,"istp":true}
[    16]@2021-05-10<UTC>19:32 >> {"r_id":24,"ct":2559,"ctp":2709,"t2r":0,"istp":true}
[    17]@2021-05-10<UTC>19:32 >> {"r_id":26,"ct":2559,"ctp":2710,"t2r":0,"istp":true}
[    18]@2021-05-10<UTC>19:32 >> {"r_id":27,"ct":2559,"ctp":2710,"t2r":0,"istp":true}
[    19]@2021-05-10<UTC>19:32 >> {"r_id":28,"ct":2559,"ctp":2710,"t2r":0,"istp":true}

with the game.tick remaining the same, while the game.ticks_played (`/time` ?) does update.

The weirdest part about pauses,

krs_rcon_fix.png (12.29 KiB) Viewed 2827 times

[boskid]

Thanks for the report, however this is Not a bug.

In order for the Rcon to work properly, its commands have to be processed as soon as possible. By "polling" i assume you are running "\c" commands and they have to be processed as InputAction because they are distributed to all connected clients and they are persisted for the purpose of replay (if enabled). InputActions have to be assigned to one particular tick and to obtain the results of running such command the input action has to be processed by the game state which requires running an update. Update is atomic as you cannot run part of it, throw some input actions to get results, throw more input actions in the same tick based on the previous results and continue. Because all of that, everything works exactly as designed for the implementation to be reasonable. Having commands in queue overrides server pause. Once they are processed, pause is effective again.

You can use the tick_paused and its a valid solution, however it does not make a game to pause entirely: it simply prevents some parts of update happening (machines are not crafting, fluids are not moving) but there are still some that are still running (like editor controller is able to move) - There are 2 tick counters, one for how many update ticks passed and one for how many game ticks passed.

[KeepResearchinSpoons]

[edit: this reply was written before the aforeplaced reply post, the nature of the forums... #include a grain of salt, since most parts of it should be obsolete]

spoila

rcon wise:

The thing about r_id:
factorio sometimes sets rcon id of the packet to the one of the caller, but usually it1 does not.
So for matching them up on the sender side it is great to have rcon return the request_id with the data, so that you call it with one and it returns one back with the data.
in case of "Cannot execute command. something" you have to rely on the timeout and drop those as failed ones.
Rcon does drop some on the factorio side. Well with extensive test polling that was to be expected tho...
Sometimes all 4 echoes per tick succeed and sometimes 1 of 2/per tick sent fails miserably.
Thus matching-by-custom-id is very convenient!


Logic wise:
I do get the raw idea behind pushing a "tick" further with an rcon event being recieved: you need to propagate the changes of one client (host) to all the peers.
However I DO NOT get at all why getting some data from a server actually

not only
updates the game state,
well, if it is not a stats read, but an upd, it does some updates, ok,

but also
makes a full blown tick simulation? eh...

Even further.
If the game has been paused manually (yesh, good luck pausing via lua)
(it was a deliberate action, by a certain admin, who had the rights to do so and had his reasons)
The rcon then comes and like,

paused? I am rcon! No rcon cares for the game being paused!
And then runs a tick.
Which it does not exactly being expected to do in the first place,
And even less so when a pause is being in effect.

Being this a manual one or being this an auto-pause,
The later being the reason of the leap-seconds for all the servers for quite some time...

[KeepResearchinSpoons]

In order for the Rcon to work properly, its commands have to be processed as soon as possible. By "polling" i assume you are running "\c" commands and they have to be processed as InputAction because they are distributed to all connected clients and they are persisted for the purpose of replay (if enabled). InputActions have to be assigned to one particular tick and to obtain the results of running such command the input action has to be processed by the game state which requires running an update.

Yay! a reply!

With this kind of a reply being expected though,

g_rcon_fix.png (14.66 KiB) Viewed 2741 times

Well a replay would probably crash on a big enough command anyways, same way tsubodai crashed it with a big enough bp string imported at once... Oh sweet it has no length limit! well replays are great anyways


I have but 3 questions left here:
1) About the current state of the pause logic:

would it be possible to somehow get
events.on_game_paused({paused_by_player:player/nil if autopause}) / events.on_game_resumed (player/nil)
that also has some form of a r/o game.is_game_paused or something
at least?

There is no way to tell if a game is "paused" currently.
Same thing about precise ups measurements.

You can guess: (somehow poll very rare, lol, ofc not possible, and guess this is 1 of 60 so -- paused! (chance is 1 of 6 among net issues and such, not bad))
But that's it.


For what is worth, lua-api tick_paused does not mention that tick_paused can actually be set to `false` when the game IS paused.
Should I report this one as a bug instead?
The bug: tick_paused is false when the game *seems* to be paused (spoiler, it is not, only some events are paused :kappa:).

From that point it works as a charm; game is not actually paused, it's just that you don't know the password or da wai to supply inputs.
And if requested for an update it runs perfectly! Cuz it is not game.tick_paused!
(which we can see)



2) is it possible to tweak the rcon logic a bit?
It is just 2 side effects that are nasty.
And aside from the pause issue the next one is the "3rd counter for the time _players_ played"

Just run an rcon command as it is NOW, but
> without force-running the simulate-entities part
(just respect the `paused_state && game.tick_paused` to decide the game.tick update really)
(if it IS necessary to run a tick while paused we can safely incr ticks_to_run or undo the eh "game_paused" state then?)
and
> without incrementing the game.ticks_played?
or, better yet, calc the map_played time without bot-updates part somehow?
I see this as an 3rd counter, since it would be just player-side updates, and not the "server-side" ones

This is actually the least effort route,
if the entities simulation would not run on pause
. (should it be
. "a dirty fix working on autopause only" +
. "another dirty fix that actually forbids the usual pause and introduces a custom one..."
. or
. "a code fix that respects game_paused"
. does not exactly matter, a "taste preference", so to say)
and
if the map_played time does not rise while it IS idle (0 players or better yet, paused state check)

then
we can run as many "full updates", as possible, as required or as necessary.
It is just that those mp-wide "updates" have terrible side effects of
> updating the entities (even when paused) and
> increasing the time_played(also when paused)

If you need to run a full logic loop for an API-read call, so be it, as strange as it sounds.
Let's call it a non-optimised API-read-write-etc call, same backend route, whatever.
just remove those side effects please and the issue of this whole thread ceases to exist.


and 3)
Is this "not-a-bug" triage a final all-things-considered official decision on the "Wube's current factorio-team" level that I can link to?
I don't exactly need a fast reply there, better yet a wording that I could report to a group of server-owners, this would be nice.
Like, the cost is too much, not a bother, can't do or something but in right terms.
If possible, before the 2021-05-15 (mid Q2, about a week ahead) or better yet a day or 2 before the line.
I would just report this as a "dead" bug that is not possible to fix both client and server side for a reasonable price,
but I need a solid confirmation, as the future resources allocation levels are somehow affected by this and I need to know for sure if this is worth it on my side as well before going into it further.
It's fine for any answer; they are not pressing the issue, just have to know for sure.



Anyways, thanks for the ride,
shout outs for
"game.encode_string"
"game.decode_string"
"game.json_to_table"
"game.table_to_json"
which proved very useful in overcoming rcon limitations!
(along with the serpent for the debug slices part)

On a side of the road,

spoiler

The lack of dynamic functions (on_tick context with 1 to 5% ups effect at best)...

make_lua_not_wars

introduced us to scenario template-compilation,
which are somehow not that bad of an idea.
It relies heavily onto `player.connect_to_server()` though.
A bit too much imo.
The main reason being you can't introduce most major changes without a server-reload.

The connect_to_server displays the "aws' " address/port
(at least not the password :kappa:)
(well since password is static you still need another session token supplied anyways)
which actually looks not as cool for the gui option,
especially since the target server is not on a public list and has a meaningless technical wild-carded domain of sorts.

However If we want to perform say "on_chunk" faster
than the client may ever be able to perform (both resource and logic wise)

welcome_to_the_server_siiiiiide

we use the server hardware and options and not the each-client-side generation,
which in turn introduces us to the on_tick "event" ques
and some peculiar bottlenecks.
Like a single rcon packet per tick to both
"read" (poll all updates, get chat, get commands, get tracked events to prevent griefing)
and
"write-update" to tiles, entities and fast ques of sounds, particles and explosion/projectiles events.

As strange as it sounds,
even with the most of it going through a global table at least 2 times
(and + 2 times the net part of the event being transmitted at least)
the overall latency and "play-value" is comparable to how the native/usual lua scenario works in mp.
(or at worst to how you drive a car in mp that is; yup a bit crappy at times you meet a net speed limit at peaks).

you_can_run_hunger_trees

And the best part is that you can even go script-intense maps like that old "hunger trees" map
without setting game.speed to 0.5 or worse
for everyone to be able to at least do something.

You just push more resources for the processing server
and choose how much wood you want
and simply let the clients and headless exchange the approaching-trees deltas only.

you_can_choose_the_next_area

The other part is that you can now have a poll on what "sector" comes next in maps like mountain fortress (or pirates, chrono and so on)
And the most interesting part is that you can "balance" difficulties and events set/events options to suit players' levels;

When you see a player that 2hours a whole game or really knows how to "game" your map (previous experience)
with e.g. mines spam
you push the "arty-strike kind of nading before the boss bitters" option into the game in turn for a challege.

This would totally wreak "your usual party" since they are yet to discover the power of the slowdown capsules
(aka yellow gas/ cheese capsules etc)
While for the good player that is the challenge they just wanted.
And can easily cope with.
(just bring a dozen of yellow capsules, and a bit more turrets lol)

While this is the line between a fun map and a dull map,
you can't exactly compile these decisions (your whole db in sorts)
onto the scenario code,
you won't do it for the personal disclosure reasons anyways.

what is left in a fully pre-determined world is a fully pre-determined adventure.

so_we_wanted_a_custom_utility_function

For new players this is too tough even on the "don't hurt me" difficulty preset,
they just need more time to adapt;
And punishing them with a restart when they just started to feel the ropes is a bit cruel and unfun.
And for the salted players it is just a scenario/game they already played and "gamed" out.
They know a strat. And they know what to expect of their peers and just ask "nades" or something and go for a nest with a "random" party
your "usual" party won't bring fish, or would decide that they are fine without a steel armor, which is ... hard to deal with without a rage-quit and dont-return-for-a-week sometimes.


For example, a bit earlier, when the Mountain Fortress was in its prime,

mfexample

When you had 10+ players that KNEW what to do,
even the hard preset was fine and oil-clicking. They pushed north, defended the train and LEISURELY crafted rocket science.
(discussing whether they need a second silo wagon or just some smelting warehouse)

and when you had 20+ new-to-mid level players, the diff function started to kick in (the player number)
While the cooperation between (!) (the hardest part in those maps, since no one "guides" players on how to do it)
and the overall progress were mediocre at its best...
... on an easy preset.

It is not as hard, but on the build-inside-a-train map all errors and skip's pile up and then snowball with players skipping whole areas since no outpost was built.

To combat this you want to have the pressure levels adjusted, right?
For a simple setup you can utilize the utility function of pollution produced!
It is really good for a production map, you have my regards for how much you can tweak bitters and pollution options for your needs and worries,

For a scenario that relies on going to a point A or B or C etc and heavily introduces manual crafting (for the cases when you want to go even fashter) however,
It doesh not work.

so_we_ve_made_a_custom_utility_function

we introduce an utility function that uses TTA (time to arrival) to an area of the checkpoint on the per-player basis,
add some crafts speed and the inventory "fullness" (and "crappiness" function over a not-required items for this area list) aside from your usual kill counts and buildings places (and "how much and how fast the buildings are placed around on average" for an area).

Currently it is a fully dedicated server-side module (albeit not fully utilized)
and I see no option to convert it to on-each-player-at-each-player
in a "sandboxed" environment that you can't really make requests from.
... or like use popular libs for ml and basic stuff of hundreds mbs in raw total ... at each client? lol, not a chance even with a good packer..

And you know... it does use rcon as a transport.
A lot of it in fact.

Also,

we_faced_and_braced_some_engine_limitations

Due to a random order of events inside a tick just about every interaction aside from the "fast" ones takes 3+ ticks per event.
Due to a nature of some apis (like, would be great to freeze-unfreeze tiles in the area to swap tiles into chunks way faster, speak about 2 layered setups with stone paths for the "recommended way" or just your average big number of tiles or even entities in general) we expect most "fat" operations to resolve in 7 to 15 ticks and long "bg" ones to span over up to 3 seconds long *which means you have to que the next ones way beforehand with the confidence as with the suicide landing burn. THIS ALONE requires rapid ups checks to ease the things in ~0.2 of a second if we went waay overboard for some reason. Happened a bit on simple tests till now, no one is perfect.
Due to a determination for all clients we have to emulate lazy-loading over these "events" as well.
There is also "items-on-belts in an area", which we want to track but do not, since it is costly. (spoiler: so we pluck inside railroads and enforce trains where possible)

See, the story is mostly fun, but the point with the rcon not being able to read some states reliably (like the is-game-paused one) or fast enough (iterate over entities collection) really bitter'ed bugged me out.

If this would be possible to fix, it's great. If it is not, then it's definitely not one if the scariest issues we've had to overcome in this way or another...

like

Imagine a whitelisted with 1-player server that is dynamically started from scrap each time, just because the info you put into global to make it work is "shared" between all peers and can even be saved onto the save file to persist. So no passwords (short tokens at best, we need them) no personal data like some of the stats, no ip lists in globals and so on.
The worst of those is the in-game friends list that had to die on the day of being born. Possible in private rooms it simply has no meaning there.
Like, the secure local storage for each of the players would totally be awesome, but wait are we inventing the chrome wheel and so on.

But well, if it is not worth to pump the thing past its boiling point and it is not worth investing into this part of the project past a certain limit -- It is fine we understand :pun-non-intended: :> and would simply adjust our plans accordingly.


Best regards, and best of luck,
youh pasher by factohian,
/ ... / ... / ... /
:>

[boskid]

would it be possible to somehow get events.on_game_paused({paused_by_player:player/nil if autopause}) / events.on_game_resumed (player/nil)

It is something that is possible to be implemented but that is a modding interface request. It would need the server to inject some extra input actions when the game is paused and resumed, but this feature has quite low value added because in almost all use cases from the perspective of the scripting, it would be visible as pair of on_game_paused + on_game_resumed events called in the same tick since "game paused" state is one where no update is happening. The only case where it would differ would be literally this one case where RCON forces game to update to execute the command. If the only thing you would like to do with those events would be enabling tick_paused, you can do some workarounds for that, for example include the `game.tick_paused = #game.connected_players == 0` in the command sent to the server.

For what is worth, lua-api tick_paused does not mention that tick_paused can actually be set to `false` when the game IS paused.
Should I report this one as a bug instead?
The bug: tick_paused is false when the game *seems* to be paused (spoiler, it is not, only some events are paused :kappa:).

For the brief moment when the server command is executed, the game is not considered paused so there is nothing wrong here. There are 2 types of pause: "game pause" and "entity pause". First one prevents update from happening at all, second one reduces update to the reasonable minimum which still allows players with editor controller to make changes in the world. Entity pause is exposed to lua because mods may also pause the entity update when showing some guis (i think Helmod is able to do something like that). Game pause is not exposed because game state is not aware of it as it is literally increasing time between updates for undetermined amount of time.

Just run an rcon command as it is NOW, but
> without force-running the simulate-entities part
(just respect the `paused_state && game.tick_paused` to decide the game.tick update really)

"paused_state" (game pause) is not part of the game state so it cannot be used to decide if the update should run the entity update or not. To make a paused_state part of the game state, server would need to inject some extra input actions and that would be part of the modding interface request mentioned previously.

If you need to run a full logic loop for an API-read call, so be it, as strange as it sounds.
Let's call it a non-optimised API-read-write-etc call, same backend route, whatever.
just remove those side effects please and the issue of this whole thread ceases to exist.

"/c" commands run lua code which is able to mutate game state so it never is a clean "read" access. Multiple commands while being executed will trigger some extra lua events like defines.events.on_console_command which may mutate game state even if you think your command is doing only a read. Trying to fulfill your use case would break other APIs.

Is this "not-a-bug" triage a final all-things-considered official decision on the "Wube's current factorio-team" level that I can link to?
I don't exactly need a fast reply there, better yet a wording that I could report to a group of server-owners, this would be nice.
Like, the cost is too much, not a bother, can't do or something but in right terms.

It is a "not a bug" because this behavior is intentionally here to make RCON work at all when the game is paused. If it would not be intentional then it could be a bug. Every topic can be discussed further. Those extra events could be implemented but that is modding interface request.