How to handle long running functions?

[hyperthalamus]

Hi everyone,
I'm new to modding Factorio (and modding in general), but have a lot of experience with software development. Super excited to get started on modding Factorio, I have played for around 800 hours and getting into mods feels like the next step in automating everything

I'm still trying to wrap my head around some of the basics regarding modding, so please bear with me. Essentially I'd like to understand how to best tackle code that needs longer than 1/60 seconds to run. From what I've understood and observed, the game will start to lag and slow down if I do any performance-intense computations in the on_tick handler.

Now from my general programming experience, I would to start a background thread and regularly communicate with that thread. But from what I've understood so far from reading tutorials, docs and other posts, this would probably cause multiplayer desyncs and therefore multithreading is not enabled? My second idea was be to use coroutines, which would allow to build concurrency without multithreading, but apparently these are also disabled for the same reason?

Now I'm wondering how to generally tackle these kind of things. I've looked at a couple of mods and it seems that many mods basically implement state machines that handle a single state per tick and then do a state transition for continuing on the next tick. But that still requires you to decompose your code into meaningful "steps", right? Are there any tips on that? Also how can I ensure that my steps actually complete within a single tick's time. Are there any approaches of tracking the time during your code and deciding whether you can continue doing stuff, or whether you should suspend and wait for the next tick? Are there any libraries that can help with monitoring/debugging these runtime/latency things?

[robot256]

The Factorio mod engine is essentially a real-time operating system. Most code is executed in response to events (interrupts) and can interrupt each other when specific actions trigger more events. Intensive computations need to be queued by event handlers so they can be executed in small chunks by on_tick, like how an RTOS enforces time slice limits for each task. This can be done with a finite state machine structure, or a subtask queue structure.
For example, if 100 chests need to be checked for new contents, check the next ten chests every tick. The state needs to be saved in the "global" table between ticks, so that it is preserved in the save file if the game is saved mid-computation.

[Qon]

As robot256 said: Queue things in the global table and work off a manageable slice each tick or once every few ticks. It's also possible to dynamically remove the on_tick handler when the queue is empty, but that requires that you make sure to add it back when mod loads if there's anything in the state that would have added it, to avoid desyncs (save state doesn't contain which event handlers are added to the events so you have to do that yourself).

https://mods.factorio.com/mod/DoRobotsB ... aticTrains has dynamic on_nth_tick handler.
https://mods.factorio.com/mod/ChunkyChunks has a work queue for adding and removing line objects over several frames.

[Honktown]

A consideration is Factorio's api is structured rather explicitly, and one may have to change expectations on how to accomplish some task. An example might be affecting every X that exists to do Y. Conceptually, your mod can be added to a running game, and you cannot get a "all-of-X" iterator. The pattern *necessary* would be to find all that exist "on_init", and work on them then, or queue them up later. Then, any new Xs only happen "one at a time" during the game - notice a clear separation of potentially game stopping (once during init) vs smaller additions?

There are often better and worse ways to accomplish some task, along with how difficult/annoying it is or how much code it can take to get slightly better performance. Examining other mods and talking with more experience modders can help there.

[hyperthalamus]

Thanks everyone, these are some good replies. I already thought it might go into this direction. So essentially it means a lot more work for us mod developers to take care of appropriately splitting the code into small processing chunks... this perspective is something new for me and I'll have to wrap my head around it.

How can I inspect how long my code for each state-machine state will run, or where bottle-necks are? I saw that I cannot access the system time from Factorio's lua? Are there any good debugging/development tools?

[robot256]

The debug overlay displays (F4 to select and F5 to enable) can show the min/max/mean execution time for each mod's scripts. We use that all the time to see what us affecting UPS, whether it is a mod or a base game function. There may also be a more detailed profiling mode I've heard of, not sure how to get to it.

If you want ideas for how to approach a specific task, we can help with that too!

[Qon]

If you want more detailed profiling the Factorio Mod debugger by justarandomgeek might be able to do that. I haven't used it myself, the debug timing output for the whole mod has been enough so far for me and doesn't need any external tools.

But I suggest you start making mods and see what kinds of issues you encounter. Often the problem isn't really that you need to profile a part to optimise it, you need to understand what parts the (mostly event) API gives you quickly and what things you have to keep track of yourself (which will be slow and might be unfeasible if not done very well) and no amount of profiling can change the API. So you need to first grip and understand the API as it is and what can be done with it.

[Pi-C]

How can I inspect how long my code for each state-machine state will run, or where bottle-necks are?

Check out the LuaProfiler! It's a vanilla tool, you can create profilers using game.create_profiler().

I'm not sure how expensive it is to run such profilers. You probably don't want to have them running all the time in a mod that has been published. So I'd activate a debugging mode (e.g., use a setting or let it depend on a variable that you set to false before uploading) and only create/poll the profilers when your debugging mode is active.

Are there any good debugging/development tools?

Since I've first noticed the Lua API global Variable Viewer (gvv), I can't imagine control stage scripting without it! Just add a "require" line to control.lua, and you can check your global table and your mod environment.

[boskid]

Now from my general programming experience, I would to start a background thread and regularly communicate with that thread. But from what I've understood so far from reading tutorials, docs and other posts, this would probably cause multiplayer desyncs and therefore multithreading is not enabled? My second idea was be to use coroutines, which would allow to build concurrency without multithreading, but apparently these are also disabled for the same reason?

Starting threads in general programming may be simple to solve many performance problems, but Factorio's modding API specifically does not allow that. There is one thing which is far more important than performance, and it is a determinism. Architecture of a multiplayer is that we only ever send input actions (things related to the players doing stuff) to the server and then back to all clients and rely heavily on the fact that server and every client will apply those input actions to the game state at the same tick and those changes will give exactly the same state on all instances of the game. When writing a mod you are not simply writing a program which has to produce any result. Your mod must provide exactly the same result on all of those instances running simultaneously without exchanging any extra data between them. Since we do not send any script data when game is already playing (they are only send during the initial map download), a mod must solely rely on all values obtained from the game state as being the same on all instances and part of this guarantee is that mod must also be deterministic.

Here there is the most important concept to understand: "Part of game state". Whenever you see "part of game state" that means its a variable which can be obtained on all instances and it has the same value on all of them. Under assumption of a program determinism, if your script is deterministic and uses only data that are part of game state, you can make changes in the game state keeping the invariant safe. If it would not be maintained, a client will desync and a player will be unhappy way more than if a game would be running slow.

We do not expose how long is a function running because that depends on the CPU speed and many other factors and as such will be different from client to client. If you would write a script that performs some heavy computation and then adds an entity to the game, a client with fast CPU would perform more computation during a single update and would build more entities while a client with slower CPU running independently in parallel would put less entities on the surface. That means the amount of entities on the surface is now different and one of the clients will desync (literally saying: game state after the update is different) and as such cannot continue as the state of the entire game will keep diverging more and more.

Threads/coroutines are not allowed. There is already a lot of extra stuff we need to take care with mods since literally every lua event could destroy everything on the surface and when the event is raised in the middle of some action, that action has to be able to deal with anything in the game state changing state including removal of anything (up to the object which caused the event in first place). Trying to add a synchronization here would only increase the amount of pain for us, and it would also bring new challenges related to avoiding desyncs. If a thread/coroutine would finish earlier on one client than on another, it would make game state differ, which is also a desync.

You may notice in almost all lua-api, that we never expose anything that is not part of the game state just for that reason: if there would be anything exposed which is not part of the game state, mods will use it and they will desync immediately.

A thing which is the closest one to being a "not part of game state" is the script loading. When a client joins we do not want every client to discard state of all of its mods and reload it as the newly joining client does. That means that there is one specific event raised on a client which is joining: its the on_load event. Its sole purpose is to tell the mod "you are just loaded, all your metatables are gone, all your global variables (not the table named `global`) are gone, you may want to fix them so your state is exactly the same as on the other clients". So the purpose of the "on_load" event is to notify mods that they need to do extra action to make their state consistent with other clients as that way it is easier to avoid desyncs. And then modders think they know what they are doing, they are setting flag during the on_load event, that flag alone is now non deterministic and they are surprised that a desync happened.

Primary solution for the long running functions is to make them incremental in any way possible. If you know that you have a lot of stuff to compute you need to spread the computation along multiple ticks. There are many catches here because what if your intermediate state is saved to save file, then a player updates mod where your new implementation is included and that implementation has new logic and yet you need to migrate from the old data structures where the computation is half completed. If you accept that then your computation may be spread across multiple ticks (worst case starting the computation from scratch in case of a script change). Simplest solution may be to fine-tune the parameters of how many steps of your logic you want to do in a single tick. If you perform too little then the response time will be greater but cpu usage will be smaller. If you perform too much a response will be fast but the cpu usage will spike. Many mods are exposing those kind of variables in their mod settings so a player can fine-tune them, up to changing them run time (as those changes are done through input actions which guarantees all instancess will get the same new value at the same tick).

[hyperthalamus]

Thanks for all of your hints and tips, they are super helpful and insightful!