Friday Facts #82 - Optimisations

Regular reports on Factorio development.
Martc
Long Handed Inserter
Long Handed Inserter
Posts: 85
Joined: Wed May 21, 2014 8:20 am
Contact:

Re: Friday Facts #82 Optimisations

Post by Martc »

slpwnd wrote:The original idea here was to simulate different chunks in different threads. However this is quite tricky to do properly - namely because overlaps and special cases. Another approach we have been thinking about recently is to do some "thematic" parallelization. Things like electric energy propagation, path finder, rail finding.
I think you can simulate factories that aren't directly connected (only with trains or robots) in different threads. But I know, finding "not connected" factories can be quite challenging, because wagons can be used as storage box.


And I have question about Steam integration, do you plan support Steam Workshop for mods? And maybe for blueprint sharing?

McRib
Inserter
Inserter
Posts: 39
Joined: Mon Apr 20, 2015 6:12 pm
Contact:

Re: Friday Facts #82 Optimisations

Post by McRib »

The list isn't so scary after all, so my optimistic estimate for 0.12 is 1.5 months from now, while slpwnd estimates 2.5 months from now. I know he is going to be right, but I still prefer to be optimistic and I also prefer to be pushed by deadlines.
You mean version 12.01 or you mean a stable version 12.xx?

FishSandwich
Smart Inserter
Smart Inserter
Posts: 1847
Joined: Sun Feb 23, 2014 3:37 pm
Contact:

Re: Friday Facts #82 Optimisations

Post by FishSandwich »

McRib wrote:
The list isn't so scary after all, so my optimistic estimate for 0.12 is 1.5 months from now, while slpwnd estimates 2.5 months from now. I know he is going to be right, but I still prefer to be optimistic and I also prefer to be pushed by deadlines.
You mean version 12.01 or you mean a stable version 12.xx?
0.12.0 will be the first version. Stable 0.12 is a lot longer away(4-5+ months?)

kovarex
Factorio Staff
Factorio Staff
Posts: 8023
Joined: Wed Feb 06, 2013 12:00 am
Contact:

Re: Friday Facts #82 Optimisations

Post by kovarex »

McRib wrote:
The list isn't so scary after all, so my optimistic estimate for 0.12 is 1.5 months from now, while slpwnd estimates 2.5 months from now. I know he is going to be right, but I still prefer to be optimistic and I also prefer to be pushed by deadlines.
You mean version 12.01 or you mean a stable version 12.xx?
12.0, after the experience we have, we can't guess the stable ....

User avatar
ssilk
Global Moderator
Global Moderator
Posts: 12084
Joined: Tue Apr 16, 2013 10:35 pm
Contact:

Re: Friday Facts #82 Optimisations

Post by ssilk »

I guess for you ( was nearly right the last time :D ): Stable 0.12 in September.

<crossing thumbs>
Cool suggestion: Eatable MOUSE-pointers.
Have you used the Advanced Search today?
Need help, question? FAQ - Wiki - Forum help
I still like small signatures...

Wolfwaffe
Burner Inserter
Burner Inserter
Posts: 14
Joined: Sun Feb 01, 2015 4:25 pm
Contact:

Re: Friday Facts #82 Optimisations

Post by Wolfwaffe »

Steam integration
This is not technically part of 0.12 as we plan to go to the steam when the 0.12 is stable, but this time we decided to postpone it no more.
Finally! Been waiting for this forever. One question though, will there be byers tiers like game has right now (belt repair man and others). Kinda wanted to get highest tier, 'cause i like the game, but i'm not really fond of messing around with paypal >>

User avatar
cpy
Filter Inserter
Filter Inserter
Posts: 811
Joined: Thu Jul 31, 2014 5:34 am
Contact:

Re: Friday Facts #82 Optimisations

Post by cpy »

There is no such thing as stable multiplayer :D
Especially with mods, and i don't play without them.

roy7
Filter Inserter
Filter Inserter
Posts: 334
Joined: Fri Dec 12, 2014 4:24 pm
Contact:

Re: Friday Facts #82 Optimisations

Post by roy7 »

slpwnd wrote:Yes. The goal is to make it playable even with 1s latency. However the latency hiding won't be perfect. This involves some dirty internal tricks, therefore we decided to go for most common actions. Things like moving, building, mining, etc. They will be (unless there is a collision) instantaneous. However in non-latency hidden actions the lag will still be felt. More on this will be in the fff.
That sounds fantastic. It's the basic actions like moving around and chopping down a tree that make a slow connection feel laggy. With those being instant, it'll feel snappy regardless of what the lag is. Very happy to hear about this. :)

User avatar
ssilk
Global Moderator
Global Moderator
Posts: 12084
Joined: Tue Apr 16, 2013 10:35 pm
Contact:

Re: Friday Facts #82 Optimisations

Post by ssilk »

In the FFF the whole crew is mentioned.
The whole?
No!
A little hedgehog is missing! See https://www.factorio.com/team

So, what has happened to the hedgehog?
Cool suggestion: Eatable MOUSE-pointers.
Have you used the Advanced Search today?
Need help, question? FAQ - Wiki - Forum help
I still like small signatures...

FishSandwich
Smart Inserter
Smart Inserter
Posts: 1847
Joined: Sun Feb 23, 2014 3:37 pm
Contact:

Re: Friday Facts #82 Optimisations

Post by FishSandwich »

ssilk wrote:So, what has happened to the hedgehog?
He got squashed trying to cross the road.. :(

Why? He wanted to show his girlfriend he had guts..

slpwnd
Factorio Staff
Factorio Staff
Posts: 1835
Joined: Sun Feb 03, 2013 2:51 pm
Contact:

Re: Friday Facts #82 Optimisations

Post by slpwnd »

ssilk wrote:In the FFF the whole crew is mentioned.
The whole?
No!
A little hedgehog is missing! See https://www.factorio.com/team

So, what has happened to the hedgehog?
That webpage needs some serious update=) But the hedgehog is still with us! Well originally there were three. Now only one is left in the office - the small one and he is sitting on my monitor watching me if I do the job well=)

bulldog98
Long Handed Inserter
Long Handed Inserter
Posts: 74
Joined: Tue Apr 08, 2014 6:40 am
Contact:

Re: Friday Facts #82 Optimisations

Post by bulldog98 »

cpy wrote:I wonder if main thread threading is planned, because you could build 4x ish bigger factories on quadcore cpu. :)
The only thing that comes into my mind for this is the following:
Image

slpwnd
Factorio Staff
Factorio Staff
Posts: 1835
Joined: Sun Feb 03, 2013 2:51 pm
Contact:

Re: Friday Facts #82 Optimisations

Post by slpwnd »

bulldog98 wrote:
cpy wrote:I wonder if main thread threading is planned, because you could build 4x ish bigger factories on quadcore cpu. :)
The only thing that comes into my mind for this is the following:
Image
Brilliant :)

User avatar
cpy
Filter Inserter
Filter Inserter
Posts: 811
Joined: Thu Jul 31, 2014 5:34 am
Contact:

Re: Friday Facts #82 - Optimisations

Post by cpy »

Hmm, i think some scripts can definitely be threaded, not necessarily useful in normal game but i'm sure mods could benefit if they could multithread some scripts.
Like treefarm, you don't need to check trees in any particular order if they want to grow, you can check them like 16 - 32 at once with threads. I'm sure there are other scripted mods that could use it.

orzelek
Smart Inserter
Smart Inserter
Posts: 3836
Joined: Fri Apr 03, 2015 10:20 am
Contact:

Re: Friday Facts #82 - Optimisations

Post by orzelek »

cpy wrote:Hmm, i think some scripts can definitely be threaded, not necessarily useful in normal game but i'm sure mods could benefit if they could multithread some scripts.
Like treefarm, you don't need to check trees in any particular order if they want to grow, you can check them like 16 - 32 at once with threads. I'm sure there are other scripted mods that could use it.
Threading of those would have quite complicated consequences for determinism of game. It could be tricky to make sure that they are threaded in exactly same way on all clients.

kovarex
Factorio Staff
Factorio Staff
Posts: 8023
Joined: Wed Feb 06, 2013 12:00 am
Contact:

Re: Friday Facts #82 - Optimisations

Post by kovarex »

orzelek wrote:
cpy wrote:Hmm, i think some scripts can definitely be threaded, not necessarily useful in normal game but i'm sure mods could benefit if they could multithread some scripts.
Like treefarm, you don't need to check trees in any particular order if they want to grow, you can check them like 16 - 32 at once with threads. I'm sure there are other scripted mods that could use it.
Threading of those would have quite complicated consequences for determinism of game. It could be tricky to make sure that they are threaded in exactly same way on all clients.

More than tricky :)

ratchetfreak
Filter Inserter
Filter Inserter
Posts: 950
Joined: Sat May 23, 2015 12:10 pm
Contact:

Re: Friday Facts #82 - Optimisations

Post by ratchetfreak »

orzelek wrote: Threading of those would have quite complicated consequences for determinism of game. It could be tricky to make sure that they are threaded in exactly same way on all clients.
Threading and determinism don't go hand in hand.

Allowing parallel loops would be nice though, and it would allow better parallelization. For example ticking all assemblers

Code: Select all

void tick(float powerfactor){//is the % of power available to allow slower progress
    if (inprogress){
        //update progress bars depending on power factor

        if(progress==100){
            //add result to output
        }
    }
    if(hasEnoughIngredients && outputNotFull){
        //remove the inputs

        inprogress = true;
    }
}
This is independent of all other assemblers and inserters and power shortage is handled through the powerfactor parameter after precalculating the power for all machines.

In the lua engine you could say, given this collection of entities run this code once for each of them. With the restriction that the function is only allowed to access that entity and read-only basic values. Adding a reduce (adding the results of the function together) on the results is a simple followup.

kovarex
Factorio Staff
Factorio Staff
Posts: 8023
Joined: Wed Feb 06, 2013 12:00 am
Contact:

Re: Friday Facts #82 - Optimisations

Post by kovarex »

ratchetfreak wrote: In the lua engine you could say, given this collection of entities run this code once for each of them. With the restriction that the function is only allowed to access that entity and read-only basic values. Adding a reduce (adding the results of the function together) on the results is a simple followup.
This is quite abstract already even without talking about the details, and preparing the restriction specification would almost surely take more time than the time saved by the loop here. In lua it would be even more complicated, as the lua engine is not prepared for parallelism even remotely. And even if it would, the modders would make faulty parallelism code almost for sure, which would lead to huge amounts of desync reports that we would have to solve.

The result for me is, that custom parallelism written in C++ (by us) might happen in the future for specific tasks, but lua parallelism isn't really an option.

ratchetfreak
Filter Inserter
Filter Inserter
Posts: 950
Joined: Sat May 23, 2015 12:10 pm
Contact:

Re: Friday Facts #82 - Optimisations

Post by ratchetfreak »

Depending the the setup that's used I can see something like the following for the parallel foreach:

Code: Select all

parallelFor(entitiesToProcess, 
            [TempEnginePool]()=> TempEnginePool->pop(),
            [&Staticinputs](Pair<Entity*, Vector<LuaCode> e, LuaTempEngine* le)=>{
                le->prepEngine(e.first, Staticinputs); 
                foreach(LuaCode code in e.second)
                    le->run(code);
               le->propagateChangesToEntity(e);
            }, 
            [TempEnginePool](LuaTempEngine* le)=>TempEnginePool->release(le));
where
  • the first parameter is the collection of entities,
  • the second is the per thread init to get a temporary lua engine from a pool,
  • the thirds is the per entity code to set the engine's globals as needed and run the code and finally propagate the changes to the entity back
  • the final one is per thread cleanup to release the engine back into the pool
This will need a global lock on the map though.

In lua code you can submit the list of entities and the fileID with the code to run (to reinforce that it's not executed in the same scope), this list may not contain doubles and optionally a reduce operation+callback. The code will then be appended to the vector paired with the entity in the entitiesToProcess list.

The loop will not be executed until all mods' events for this tick have run and the results of the parallel code will be propagated back through the callback the next tick.

The order in the LuaCode vector will be deterministic (based on order of execution of the mods, and the order in which they pass the code), the setup disallows propagation of changes to other entities.

Post Reply

Return to “News”