Enable coroutines in data phase?

mrudat · Post by **mrudat** » Sat Dec 28, 2019 12:22 am

TL;DR

Allow using coroutines in the data phase.

What?

If re-enabling coroutines is simple, eg. omitting coroutine = nil, it would be nice to have them available in the data phase.

Why?

From the posts on the forum, I believe that coroutines were removed from the Lua interpreter as there was no way to safely serialise them in the saved game state.

Given that only data.raw, as understood by the compiled classes actually leaves the data phase, there should be no issues with using coroutines in the data phase.

They make certain algorithms easier to implement; it's entirely possible to do anything that can be done using coroutines using other methods, but they can range between somewhat and significantly more complicated to implement.

PyroFire · Post by **PyroFire** » Sat Dec 28, 2019 5:43 am

mrudat wrote: ↑Sat Dec 28, 2019 12:22 am but they can range between somewhat and significantly more complicated to implement.

Got any examples or proof of that claim?

mrudat · Post by **mrudat** » Sat Dec 28, 2019 7:21 am

For example Cartesian product in Lua, which has a functional, coroutine-based, and iterative implementation.

Of the three the coroutine-based one is simpler; you don't have as much scaffolding around the algorithm to confuse the reader about what's going on. I can just read the code, and I recognise what it's attempting to achieve; the other two implementations? not so much. Hmm. I think I may have added that example, actually, the last time I tried to use coroutines with Factorio, so perhaps I'm biased?.

For further reading the Wikipedia article on coroutines

As far as I am aware, coroutines are similar to closures (a component of functional programming); there's nothing that you can implement using closures or coroutines that not feasible to implement using straight imperative code; it's just that generally speaking, you can say things more succinctly and more simply using those features than doing without.

The Java Streams API is almost trivial to implement with the addition of coroutines but requires significant scaffolding without. JS Promises and the shiny new async/await syntax are a special case of coroutines. Generator expressions in Python are a subset of coroutines.

Edit: fix gramma.

PyroFire · Post by **PyroFire** » Sat Dec 28, 2019 8:03 am

mrudat wrote: ↑Sat Dec 28, 2019 7:21 am wikipedia and other citations

I asked for examples and proof that coroutines make certain things easier to write.
Not for links to other websites that have coroutines in their examples, as this is not proof of anything.

I want to see factorio lua code that is cleaner when written as a coroutine when compared to a simple loop.

This is setting aside the fact that, because the data stage happens in a single frame (so to speak), the benefits of using coroutines is lost entirely - you should just be using loops like for and while, or regular functions.

mrudat wrote: ↑Sat Dec 28, 2019 7:21 am As far as I am aware

Your awareness appears to be limited to wikipedia and other peoples tutorials, and not your own work.

Coroutines add zero functionality that isn't already possible, and in fact would make the lua backend more complicated for zero gain or purpose.

-1.

And for the record, coroutines actually makes things harder to write and can be more complicated (particularly if you use them properly instead of as a stand-in for a regular function)
Not easier.

furthermore, your citation on rosettacode is without merit.

https://rosettacode.org/wiki/Cartesian_product_of_two_or_more_lists#Lua wrote: Using coroutines
I have not benchmarked this, but I believe that this should run faster

Wonderful, so you're basing this idea on something that literally, and openly admittedly hasn't even been tested.

And as a final nail in the coffin;

mrudat wrote: ↑Sat Dec 28, 2019 7:21 am As far as I am aware, coroutines are similar to closures (a component of functional programming); there's nothing that you can implement using closures or coroutines that not feasible to implement using straight imperative code; it's just that generally speaking, you can say things more succinctly and more simply using those features than doing without.

Do you even understand what coroutines are actually used for?

I'm guessing not so let me explain it.

Coroutines can be thought of as setting up functions to run on multiple CPU threads at the same time (although lua doesn't actually operate that way) - the coroutine can be halted or resumed at any point, and maintains a persistant lua environment (meaning variables and values are kept between yields and resumes, which can be useful for delayed iterative processes, but it is still more complicated)

It's been a while since i've used them, but this is the rough shape you need.
For example,

Code: Select all

local function DoCoroutine(varg)
	local cheese=1
	coroutine.wait()
	game.print(cheese) -- 1
	game.print(varg) -- "something"
	coroutine.return(5)
end

local myco
script.on_nth_tick(1,function() myco=coroutine.start(DoCoroutine,"something") end)
script.on_nth_tick(2,function() game.print( coroutine.resume(myco) ) end) -- prints "5"

Since we're talking about the data stage, let me give you an example that doesn't use on_nth_tick (control stage)

WITH COROUTINE:

Code: Select all

local function DoCoroutine(varg)
	local cheese=1
	coroutine.wait()
	game.print(cheese) -- 1
	game.print(varg) -- "something"
	coroutine.return(5)
end

local myco
myco=coroutine.start(DoCoroutine,"something")
log(coroutine.resume(myco))

WITHOUT:

Code: Select all

function DoWithoutCoroutine(varg)
	local cheese=1
	log(cheese)
	log(varg)
	return 5
end
log(DoWithoutCoroutine("something"))

Yeah... certainly looks easier to use coroutines than not use them. *cough*

I ask you again,

PyroFire wrote: ↑Sat Dec 28, 2019 5:43 am
mrudat wrote: ↑Sat Dec 28, 2019 12:22 am but they can range between somewhat and significantly more complicated to implement.
Got any examples or proof of that claim?

mrudat · Post by **mrudat** » Sat Dec 28, 2019 9:26 am

Um. It's not possible to have prior examples of coroutines for Factorio use-cases because you can't have a working example, because coroutines are disabled?

After having thought about things a little, I'm fairly certain that the rosetta code example is something I actually did write for Factorio and posted it there so that I wouldn't have completely wasted the time in constructing it. Also, the code on rosetta code has been tested, in that it passes the provided test-cases; it's just not been benchmarked (which is literally what the comment says).

I happily admit that I don't have a good understanding of all of what coroutines are, as Lua is the first language I've used that has native coroutine support, and the majority of my Lua coding of late has been for Factorio. That said, I've worked enough with generators and green threads in other languages (both of which are subsets of coroutines) that I have at least some idea of what coroutines can buy you. What I'm missing is the feature space that sits outside generators + green threads, which I seriously doubt I'm going to be exploring in Factorio. I'm fairly certain you need to be doing asynchronous stuff for the more interesting bits about coroutines to become relevant, which is a Shouldn't Happen in Factorio.

Mostly what I'm hoping for is that restoring coroutines will help write cleaner, simpler code (even if the implementation is less than simple). If it happens to go faster, that's a happy accident, given that, on average, developer effort is more expensive than compute time.

I fully agree that it also can help you write baroque code where you need to draw yourself a map over a period of weeks to figure out what it's actually trying to do, but you can do that with any language or feature. My first language as far as code goes is PERL, so I'm familiar with a language that enhances inscrutability.

I'd happily settle for generators, except that Lua doesn't have generators, because it already has coroutines, so generators as a core language feature don't exist, but coroutines are disabled in Factorio, so we can't have generators in the data phase where they might be useful.

Edit: added missing.

mrudat · Post by **mrudat** » Sat Dec 28, 2019 10:12 am

I did some searching. I wrote a mod that I haven't published as it's kind of fragile, that calculates crafting recipes that consume and produces barrels of fluid while also applying productivity (in the form of productivity modules used as catalysts).

I wanted to calculate the cartesian product of barrels of different sizes (in case there's more than one kind of barrel for a given fluid), crafting recipes, and combinations of modules. I looked at the functional and iterative solutions and thought that they were more complicated than was strictly necessary, so decided to write a clearer version that uses coroutines as a generator, which I then discovered won't work in Factorio, so I ended up testing it in an external Lua interpreter and posted it on rosetta code.

Post by **ssilk** » Wed Jan 01, 2020 1:20 am

mrudat wrote: ↑Sat Dec 28, 2019 10:12 am I wanted to calculate the cartesian product of barrels of different sizes ... I looked at the functional and iterative solutions and thought that they were more complicated than was strictly necessary, so decided to write a clearer version that uses coroutines as a generator, which I then discovered won't work in Factorio, so I ended up testing it in an external Lua interpreter and posted it on rosetta code.

Which problem do you want to solve?

Coroutines are useful, if you have a problem, that takes so long to calculate, that it makes sense to solve the problem on more CPU's.

Edit: A bit less lines of code versus a clean, simple startup without side-effects?

Honktown · Post by **Honktown** » Tue Jan 07, 2020 7:30 am

I think a tl;dr opinion most people would have on this is "why?". The data stage can take as much time as you want (and it really shouldn't - nothing's happening). The normal "best case" for coroutines is complex behavior that can change function context without function call overhead, buuuuuut the data stage isn't doing anything. You're not going to save any worthwhile performance by adding coroutines. I'd be interested in an algorithm implementation that's A) awful in normal language constructs and B) is something actually useful to do in the data stage that makes you want these. Lua has dynamic function support, which I don't recall mis-using to my own ends during the game, but I certainly was thinking about it (using it to make one-time functions that stop existing after being ran).

Edit:

My first language as far as code goes is PERL, so I'm familiar with a language that enhances inscrutability.

Lua sounds a bit like Perl when you're reading about it, but damn if they don't have a thing in common (except a table is almost like a hash. almost).

Got bored and implemented a looping function:

code

result

What's wrong with old-school non-fold/map/whatever code?

Post by **Rseding91** » Sat Jan 11, 2020 7:53 pm

I disabled coroutines completely from the version of Lua we use (just #ifdef-ed them out of the binary). They aren't coming back.

golfmiketango · Post by **golfmiketango** » Sun Jan 12, 2020 7:48 am

It's mostly just syntactic sugar, right? Hopefully it's for some kind of good technical reason but somehow I have my doubts.

As for a use-case, that's easy. Your mod needs to check 100,000 modded entities for some complicated condition every two minutes. It's going to take a long time; you'd better break that s**t up, right?

With co-routines, you can iterate over the entities, like a normal programmer would want to.

Code: Select all

for drill in drills: update_drill(drill);

Without co-routines, you are stuck implementing some kind of crude async framework, from scratch, as procedural code, to get the job done. i.e.:

Code: Select all

procedure drill_update_work_this_tick;
var i;
begin
  if game_tick % 600 == 0 then
  begin
    for i := 1 to (total_drill_count / 12) + 1 do
    begin
      last_drill_analyzed += 1;
      last_drill_analyzed %= total_drill_count;
      update_if_drill_needs_blinking_light_animation_or_whatever(
        modded_drills[last_drill_analyzed]);
    end;
  end;
end;

Of course that's the simplest case: what if the drills need to be updated only after we update any drill-grinder-monkeys that will be cranking them if and only if they are within 100 foot radius, during the daytime or in a highly illuminated area... etc

The inevitable end-game: people get sick of writing that type of code, over and over. There will be less mods. Or, some marvelous savior-type gets to build some kind of framework to solve the problem at scale.

What might that look like? Well here's a lovely example from python:

Code: Select all

def wrap_future(future, *, loop=None):
    """Wrap concurrent.futures.Future object."""
    if isfuture(future):
        return future
    assert isinstance(future, concurrent.futures.Future), \
        f'concurrent.futures.Future is expected, got {future!r}'
    if loop is None:
        loop = events.get_event_loop()
    new_future = loop.create_future()
    _chain_future(future, new_future)
    return new_future

That's from asyncio.futures.

Enjoying the beautiful simplicity? Never fear, there's a total of:

Code: Select all

cpython/Lib/asyncio $ find . -name '*.py' -type f -print0 | xargs -0 cat | wc -l
13071

more lines of scripted code like this in that folder (the above is by no means the worst of it).

Of course python is way more crazier and feature-creepy than lua, and asyncio has tons of technical debt that a factorio framework would not.

But, in case it's not obvious, scheduling a bunch of parallel batch jobs, that are most readily conceptualized as parallelized procedural code, in a single threaded scripted environment, is why somebody would want co-routines.

Getting multiple inter-dependent queues of pending batch work to trickle through at a programmatically limited pace is a classic "easy problem to solve poorly but hard to solve well" type of problem. To coin a phrase. I guess I don't need to tell you that, devs. You guys are basically the earthly masters of precisely this task. But you're cheating, with native machine code and OS threading. Bridging that gap (partially) for toy languages is precisely the problem that co-routines solve (ameliorate is a better term I think).

I guess we're just counting on Nexella to do it... In the meanwhile, mod developers get to suck it up and code ad-hoc algorithms.

Obviously modding is totally possible in factorio without co-routines. The feature is there to help coders save mental energy for something worthwhile, instead of having to reinvent the for loop, over and over.

I understand if there's a serious technical reason it can't be done (for starters there'd be a dependency on closure persistence across saves), but if it's for some kind of aesthetic preference... I think you should reconsider, different strokes for different folks.

Factorio Forums