Treefarms and Co-Routines

[Buggi]

So having updated treefarm lite to 12.20 I've been getting familiar with the code trying to find ways to optimize it so as not to lag games.

Here's the problem, every treefarm has 49 'slots' for a seed. All the growth states of a tree are different objects as there is no in-game way for trees to grow.
Every 60 ticks (1 sec), each treefarm parses through the slots it has until it either finds a slot to update or runs out of slots.
If it finds something to update, it destroys it, and places the new item in the growth table, or it harvests the tree, or it plants a seed.

This is why in your first treefarm you see a seed planted every second or so.

The problem is, when you have LOTS of treefarms, each one has that 60 tick delay. So with 60 treefarms you would at best get one updating every tick. And this REALLY lags the game.

So I was looking into LUA co-routines. A sort of multi-threaded way to handle treefarms. And was wondering if Factorio supported LUA Co-routines.

That way, I could create a master treefarm manager who would manage the treefarms outside of the main game thread.

Thoughts? Ideas? Suggestions?

[orzelek]

As far as I know co-rutines in lua do not actually represent threads.

The finite water mod used them I think and there were some suspected issues with them so I'm not sure if they are supported.

One way that would make the treefarm faster would be to check a slot on a farm every x ticks. That would require some additional bookkeeping but would spread the load around to remove stuttering it can cause with a lot of farms. Drawback would be the fact that growing could become slightly slower due to different delays in checking of fields.

This is the method that YARM mod is using and it seems to work very well there.

[Zeblote]

I would imagine that anything you do in a sperate thread, if even possible, instantly causes a de-sync.

Maybe you can use a more intelligent data structure instead of checking all 49 slots every time?

[DaveMcW]

You could try storing the master set of objects in a lua table, and only use find_entities_filtered to delete an old tree. This should speed up the checking routine.

[Buggi]

Yeah, the more I think about it threading it would cause almost instant desyncs.

Increasing the delay, or holding all objects in a table really wouldn't work as it would still have to run eventually and cause lag. So even if it was 2 minutes, every two minutes your game would hitch.

One small thing I could do is plant a seed in the same spot after harvesting, but as I've seen with the collision boxes on trees they interfere with seed planting.

How is it the game can have thousands of things running on belts all over your base, but a few treefarms absolutely kill the game update stats?

[roy7]

What about using a concept of 'seasons', like there's planting season when you put the seeds down. So you don't put seeds down all the time. I don't mean literal seasons, I just mean as a thematic concept. Actual implementation would run something like this:

When full # of seeds reached, like inputs for a factory, plant all seeds into all slots in one step.
Now that we know we are in 'growing' season:
After tick X, update all slots with a seed to a sapling.
After tick Y, update all slots with a sapling to a young tree.
After tick Z, update all slots with a young tree to an adult tree.
After tick XX, harvest all slots with mature trees.
Repeat loop once # of seeds available > empty slots. Or harvest every tree every time, and only plant into full empty fields.

In my multiplayer base with 60+ treefarms we had to tear them all down and build greenhouses instead. However, if the main work of treefarm isn't happening every tick and all you are doing is waiting for a countdown timer to go off, that should use close to zero cpu? And the update itself is only one tick of work, similar to what treefarm does now every single tick. A single treefarm doesn't hurt the game performance that I notice enough to care about. If I make 60 treefarms and none happen to 'plant' on the same tick, then at worst in any 1 tick I'm only processing one treefarm plus 59 "is countdown timer down to 0 yet" tests.

[Klonan]

I encountered a similar problem with my wind turbine mod, it was checking all windturbines every x ticks, and was causing stutter each time it checked...

So here is the solution i came up with

Code: Select all

function check_turbines()
   if global.wind_turbine ~= nil then

  	    for k,gen in pairs(global.wind_turbine) do
      		if k % 250 == game.tick % 250 then

				if gen.valid then
         			if gen.fluidbox[1] ~= nil then 

         					local pot = gen.fluidbox[1]
							pot["amount"] = 10				
							pot["temperature"] = 100*(game.wind_speed*25)
							gen.fluidbox[1] = pot
							
					else table.remove(global.wind_turbine, k)

					end				
				end
			end
		end
	end
end

The key element to this is:

Code: Select all

if k % 250 == game.tick % 250 then

This spreads the checking on entities over the entire 250 ticks, so for each tick the work is only 1/250th of the whole table
You can change this to whatever number is suitible.

Hope this helps

[roy7]

I thought about suggesting a modulo on the treefarm # perhaps, to guarantee spreading them out, but you know what? If it takes, say, 240 game ticks (4 minutes to grow each step) and you do all of the planting/checking at the same time instead of letting seeds plant whenever, then you could do mod 240 and only update the treefarm 4 times. (Plant, grow 1, grow 2, grow 3, harvest.)

May still want to include a timer/offset so not every treefarm would always fire at the same moment. But using the modulo trick and 'speeding up' the treefarm seems like a possible winner?

Edit: Basically you already do that by k % 250. So my suggestion for Buggi to try is change 250 to the time it takes a tree to grow from one step to the next, and each update every planted tree item grows to the next step in the process and new seeds are planted. You wouldn't need to use the 'season' idea any more, since you are only working on a single treefarm in any single tick. The only noticeable change is that seeds wouldn't be planted until the next update.

[roy7]

I never knew this before looking at the code, but it appears the two seed types have different growth rates, different amounts of random variation in growing length, and different efficiency based on terrain type.

Random growth times, varying efficiency, etc makes things more complex than just reducing the # of update ticks it appears.

[orzelek]

The variation of crops and the fact you can register them is why I thought that checking it square per farm at every time interval would smooth out the time per tick used.

Only drawback I see is that with 49 squares per farm and checks every 10 ticks it would take almost 500 ticks to go over whole farm. That means that some crops could grow slightly slower then intended.

[Fatmice]

What about storing a growing table to be checked on the tick? When planting the seed, compute the next tick that it will be updated. Say planted on tick A, next update is tick A+B, where B is computed on some algorithm. Add the seed to the growth-table

Code: Select all

table.insert(growth-table, A+B, seed-info)

Then on the tick, check

Code: Select all

if growth.table[tick] then
    actions/function calls
done

So at most 1 seed is updated per tick and not a bunch of seeds causing hick-ups. If you have 12000 seeds, it would take at least 12000 ticks to update all of them, one at a time on each tick. I know there is an effect of the soil and a random number added to each seed's growth time such that it is almost impossible to get two seed to grow at the same time. Am I understanding this problem right?

Edit: alternative to table.insert is simply direct assignment.

Code: Select all

growth-table={}
growth-table[A+B]=seed-info

Either is going to result in a sparse table so do not use #growth-table to count.

[Zeblote]

How is it the game can have thousands of things running on belts all over your base, but a few treefarms absolutely kill the game update stats?

Because we don't have luajit yet. Normal lua is quite slow when compared to c++.

[kiba]

Random growth and variation causes unnecessary complexity that does not increase gameplay value.

All it does is force players to build more trees to handle the inconsistent output.

[Zeblote]

Random growth and variation causes unnecessary complexity that does not increase gameplay value.

All it does is force players to build more trees to handle the inconsistent output.

Wait, the trees don't grow with a constant speed ...?

[kiba]

Random growth and variation causes unnecessary complexity that does not increase gameplay value.

All it does is force players to build more trees to handle the inconsistent output.

Wait, the trees don't grow with a constant speed ...?

From my understanding of what was discussed, yes.

[Fatmice]

Random growth and variation causes unnecessary complexity that does not increase gameplay value.

All it does is force players to build more trees to handle the inconsistent output.

Wait, the trees don't grow with a constant speed ...?

No, it's been like that since forever. This is to give the illusion that the trees are alive.
I'm sad to hear drs9999 has stop working on it.

[Rseding91]

If you're looking for ways to optimize TreeFarm you can look at the re-write I did of it for my own game:

Treefarm-Mod_1.2.0.zip

(2.42 MiB) Downloaded 287 times

Note: this was written when there was a bug with saving sparse tables so I wrote it by converting index values to strings - don't use the code in that attached file directly - just use it as a reference and then write the correct code Also it went through *many* iterations of different attempts so the code in that zip is a giant mess.

It's an older version and I didn't put any effort into making it backwards compatible with old saves but the logic can still be extracted and applied to the current version of TreeFarm.

The primary reasons TreeFarm (now discontinued) is slow:

• The trees list to grow is acting like a deque: the first item in the trees-to-grow table is the next tree to tick. When it's ticked it is removed and inserted into the trees-to-grow at the next tick it should grow
• The farms scan the entire area for the "next spot to plant" each time they tick

Lua tables do not work well when you table.insert into the middle of them or table.remove from not-the-end of them. I can't stress this enough: every time you table.insert() into the middle of an indexed lua table it has to push all elements after the insertion point out by 1. Every time you table.remove() from not-the-end all elements after the one you remove are shifted up by one. This happens every time a tree is grown and so it eats a *ton* of CPU time.

To fix this: you can use (like I did) a sparse table of trees to grow. The "trees to grow" contains a table of tables. You index the table with the current tick to see if that index contains a table of trees to grow and if it does you grow all of those trees and then re-insert them at the next tick they should grow.

It looks like this:

Code: Select all

script.on_event(defines.events.on_tick, function(event)
  if global.treesToGrow[event.tick] ~= nil then
    growTrees(global.treesToGrow[event.tick])
    global.treesToGrow[event.tick] = nil
  end
end)

When inserting seeds into the table to grow:

Code: Select all

function insertSeed(seedTable, nextGrowthTick)
  if global.treesToGrow[nextGrowthTick] == nil then
    global.treesToGrow[nextGrowthTick] = {}
  end
  
  table.insert(globa.treesToGrow[nextGrowthTick], seedTable)
end

That way you only ever tick the trees you want to grow. Checking if you have trees to grow is a single index operation (fast), inserting is at max 2 index operations (fast) and moving seeds around between growth stages is also fast.


As for the planting of trees on farms: store the x/y of the last plant attempt and increment y each attempt. If y > a column reset it to 0 and increment x. If x > then a row reset both to 0 and restart. That way it scans the entire area 1 tile per growth attempt (or how ever many you decide per attempt) and it doesn't need to re-scan the entire area each time.


The above changes when I tested TreeFarm showed a constant 1/10th tick time as before the changes. That means that before it would take 2 MS/tick to run X tree farms and after it would take 0.2 MS/tick to run the same tree farms.

[Fatmice]

Yes, sparse table is the key to not compute needlessly.

[Buggi]

This has turned into a very helpful thread. Thanks everyone!

[roy7]

Speaking of which, is there an updated TreeFarm anywhere with Rseding's performance improvements applied?