Assembling machines and signals

[voxel]

There has been some discussion about how to make the assembling machines more usable from the circuit network.
In particular it has been suggested here: viewtopic.php?f=9&t=30853&start=120 to make it possible to set the recipe of a machine. This has also been partially implemented in this mod: https://mods.factorio.com/mods/nbp/circuit-assembler

In this thread I was going to flesh out the main ideas a bit more and add some suggestions of my own.

So the main idea is to make it possible to set the recipe of an assembling machine using a signal. The machine would simply pick the first signal or largest signal (in whatever way is consistent with how other entities work) and change the recipe to that. There remains the issue of how to deal with residual items and modules (some modules can only be used for some items). For items one could

• Throw away any items that cannot be used for the new recipe.
  This is nice because it is very simple to use and to understand. However it has obvious efficiency implications, especially when crafting very expensive things such as rockets and satellites.
• Put all items that cannot be used for the new recipe in an internal chest from which inserters can only take out items, but not insert to. Potentially this could be accessed from a specific place on the assembling machine however I think for simplicity it is best not to do that and make inserters that are trying to remove items from the machine take items from both the internal chest and the output slot of the recipe. If one wanted to separate the residuals from the output one could simply use a filter inserter with the same signal as the assembling machine anyway. The size of the internal chest would be limited to say 6 slots to prevent it being used as an arbitrarily large chest.
• When switching the recipe the residual items would be available for an inserter to remove from the assembling machine and only when all residual items have been removed, the recipe will switch completely and the machine can start to produce the new recipe. This is a nice and almost stateless approach in my opinion. It does not require any changes to the UI except possibly that the "current/capacity" part of the tooltip should be removed when it is switching to a new recipe. The mechanics would be that: an assembling machine has an inventory. Items can be inserted into the machine if and only if the item is part of the recipe for the machine (based on the circuit network) and the machine has no items in its inventory that are not part of the recipe. An item in the assembling machines inventory can be removed using an inserter if and only if it is the output of the current recipe or if it is not an input to the current recipe. This is also stable even if input signal is unstable. Say that the input signal makes the machine switch to another recipe for a single tick and the next tick it switches the recipe back again. You would likely loose the progress for the current item, but the items are still in the input slots and the machine can start to produce a new item immediately. Compare this with the previous option which would immediately when the recipe was switched place the items in an internal inventory and would have to be removed and then inserted again before they could be used as inputs to the recipe again.
• Put all items into an internal active provider chest so that logistics robots can remove it for you. Could also be formulated as the previous option. This has the downside that logistics robots are required to remove the items and it may not be obvious for the player why they can be removed using logistics robots but not using an inserter.

I would suggest the 3rd option as it requires the fewest UI changes and seems intuitively to make the most sense to me. It is important in game development to avoid making the common use cases more complex when implementing new features. This has the benefit that it would change absolutely nothing, not even the UI, when used without signals.

I would also like to make an additional suggestion. A way to get the ingredients for a particular recipe. It may seem like there are few cases where you would need this, and there probably are. However together with setting the recipe for a machine, this would allow another level of automation in Factorio, namely automatic the layout and routing of everything that assembling machines produce and consume. For larger factories, one might want to automatically transport say processing chips to the assembling machines that actually use them. This is straight forward when the assembling machines have fixed recipes as if you would need the inputs to a recipe, you can just plonk down a constant combinator next to it, however when the assembling machines can change recipes there is no easy way to gain that information so that you factory can know where to route items. For an example where this is done, take a look at the video here https://youtu.be/3gAVgGCn04w. All assembling machine units use the circuit network to "request" items from the large central storage. The system will try to send just as many items to keep the assembling machines busy (the actual logic is implemented using a PID controller, but that is not so important for this discussion). To completely automate this (which is of course the ultimate goal in Factorio) one would want to automatically set the recipes on the assembling machines based on demand and resource availability (so that it could for example automatically add more assembling machines for blue science if there are enough ingredients and the supply of blue science is running low). However, at least for this layout, there would be no way to for the factory to know where items should be routed as there is no information about what the inputs to a recipe are.

The obvious way to implement this would be to simply have an option to make the assembling machines send the ingredients for the current recipe as a signal (so if a recipe required 5 iron and 2 coal, it would send exactly 5 iron and 2 coal as a signal).
Another option would be to introduce a special combinator which for an input signal outputs the ingredients for the (first?) signal. This has problems however since a particular item may not necessarily only be possible to produce in a single way. For example the various types of oil can be produced either in refineries or using cracking, so the output of the combinator would not be easily defined. This also has the downside that it introduces a new building, which is not desirable for a simple feature like this. It does raise the question however if there should exist a better way to gain this kind of more specialized information. For example one thing one might want to do is to get the time to craft a particular recipe to for example calculate the number of items per minute that an assembling machine would need when running at full capacity. This is not something I would need at the moment, and it is just an example. However it seems like there would undoubtedly be more kinds of information that one might want to extract from the factories and it is worth thinking about how that could be exposed. Currently it would be hard to output (for example) the crafting time as well as the recipe as there would be no way to distinguish what signal is the recipe and what signal is the crafting time.

[Yoyobuae]

I would also like to make an additional suggestion. A way to get the ingredients for a particular recipe. [...] However, at least for this layout, there would be no way to for the factory to know where items should be routed as there is no information about what the inputs to a recipe are.

That sort of information is constant, therefore my first though is to program it into constant combinators.

I think stuff like the in-game video player demonstrates the huge ability to store information and access it in an orderly fashion. The amount of data needed to program all the recipes and such is several orders of magnitude less.

Now, like was the case for the video player, I'll probably load up that data using scripts rather than manually.

[ssilk]

Nice post, good thoughts, but:
Where is the TL;DR??

[MeduSalem]

Somehow 2nd and 3rd option basically sound the same:

• Items that cannot be used for the current recipe are moved to an internal residual item buffer
• Output (Filter) Inserters can take items from that residual items buffer
• Input Inserters can only input resource items from the current recipe.

The main difference is:

• 2nd option the Assembler could already start crafting the new recipe once all the resources for the new recipe are in the Assembler, even if there are still residual items in the residual item buffer.
• 3rd option CAN'T start crafting the new recipe until the internal residual item buffer is completely cleared, even if the required resources for new recipe are already in place.

So it sounds like the 3rd option is actually delaying things artificially and increasing the overhead because you ALWAYS need to wait until the residual item buffer is cleared with EVERY recipe switch, decreasing overall crafting throughput considerably if you always have to wait for the slow inserters to output everything before you can start crafting the new recipe.

I mean as I see it... the Assembler could already start working on the new recipe after moving the unusable items to the internal residual item buffer and once the actually required resources are inside the assembler. The residual items can still be removed from the buffer in the background at the same time.

Similar to how CPU pipelining works, just with a 3-stage assembly pipeline instead or so:

Pipeline.png (8.01 KiB) Viewed 10202 times

In each step the Assembler would do following:

1. Recipe Resource Fetch: Look at current Recipe signal and allow input inserters to prefetch necessary resource items (if they are not already inside the buffer)
2. Recipe Crafting Execute: Start Crafting the recipe once all required resources are inside
3. Items & Residuals Output: Output finished items. Output mismatching residuals if new recipe signal mismatches old recipe. (Basically your Write Back for those who are into CPUs)

The output can be handled by 2 different inserters... one that handles the output of finished items and one that handles the output of residual items.

This way one can make it so that there is always a limited "internal cache" for resource items... and as long as one is switching between the same 2-3 simple recipes or something there would never be residual items to be removed because they fit into the cache. But if you switch between more complex recipes or a lot of simple recipes then the cache will overflow and the oldest (least used) resources will be moved to the residual item output for pickup. And the assembler will only stall if you don't remove the residuals fast enough filling up the residual item output.

That would increase the overall throughput for simple recipes because the penalty of switching the recipe is largely mitigated.



Because with all the suggestions of "taking out residuals" almost nobody ever thought about how inefficient the throughput gets if the assembler always stalls waiting for removal of items. It becomes way too ugly that the machine spends more time waiting for item removal/insertion than it actually does pure crafting... It becomes so inefficient that it is not worth going for in comparison to dedicating machines to specific recipes.

So my above suggestion of pipelining the input/crafting/output is absolutely necessary to increase the efficiency of the entire "Set recipes"/"Smart Assembly" concept.

[voxel]

@MeduSalem
Yes, the 2nd and 3rd options are very similar and you are correct in that the 2nd option does have a higher throughput if you are switching recipes often.
As a game feature, it is however not just the throughput that should be maximized, one also has to take into account the overall complexity it adds to the game. The 2nd option would require additional UI for the storage of the residuals, which would also need additional rules that limited how much one could store in it. It also adds annoying edge cases, such as what would happen if a machine gets a signal to switch the recipe, but it cannot because the internal storage is full.
In any case you would have the time penalty of loosing the progress of the currently crafted item, so in any case you would not want to switch recipes too often since that would hurt throughput. I think the additional penalty of not being able to craft until the residuals have been removed would be a very minor issue.

@ssilk
Yeah, sorry. The post turned out longer than I initially intended.

@Yoyobuae
Yes, it is a constant, however the problem with using a combinator is that a single item may be crafted in different ways. For example light oil can be either cracked from heavy oil or produced from refineries. I suppose one could limit it to assembling machines as in that case it would work.

[MeduSalem]

Yes, the 2nd and 3rd options are very similar and you are correct in that the 2nd option does have a higher throughput if you are switching recipes often.
As a game feature, it is however not just the throughput that should be maximized, one also has to take into account the overall complexity it adds to the game. The 2nd option would require additional UI for the storage of the residuals, which would also need additional rules that limited how much one could store in it. It also adds annoying edge cases, such as what would happen if a machine gets a signal to switch the recipe, but it cannot because the internal storage is full.
In any case you would have the time penalty of loosing the progress of the currently crafted item, so in any case you would not want to switch recipes too often since that would hurt throughput. I think the additional penalty of not being able to craft until the residuals have been removed would be a very minor issue.

I think that it heavily depends wethether the assembler would otherwise be a lot of the time idle or not... and if we are conceptually designing "smart assembly" to be something that increases efficiency of idle machines or if we also want it to be a feature that let's the player adjust the available "crafting resources" (amount of assemblers/furnaces/machines) better to the actual "crafting demand" (amount of each desired item type given at each point).

If it is a lot of the time idle then a recipe switch every now and then doesn't hurt the throughput really that much because a machine stalling for a few seconds from recipe switch would still be better than a machine idling around 90% of the gametime.

But if you are going to use it let's say for smelting various ores to plates then there's probably going to be a lot of machine uptime... and then switching recipes is a huge performance hit. And as I said you might want to do that to adjust your available furnaces better to the actual proportional demand of each item.

If the later case turns out to be so inefficient that just using 2 seperate furnaces to smelt the ore dedicatedly becomes a lot more efficient (even if they are both idle at points) then it is not even worth having that feature because nobody who's somewhat into optimizing things would decrease efficiency just for the sake of having additional complexity.

So a fully programmable factory for absolutely everything like people "expect" from such a feature might actually turn out not to be worth it.

The issue becomes even more appearent if we also use Beacons in addition to such a "smart assembly". Because then recipe crafting times are so ridiculously fast that you would probably spend something like 80% of the time idle just waiting for the removal of residuals from the assembler. It becomes a true bottleneck that is just not worth going for.

There would be a point where I'd just say "cut the crap" and move back to dedicated assemblers... and in the end I would probably end up using "smart assembly" with recipe switch only for recipes that take very long to craft (like Modules) or were the demand is comparably small (like Chests or Inserters, Belts/Pipes, etc in endgame).

The faster the thing is able to craft then the more a staged pipeline becomes necessary... because I wouldn't use it if the assembler is continously stalling waiting for residual items to be removed.

I'm suggesting it because I know it will be a huge problem people might be undererstimating now but would complain about later on if it isn't considered beforehand.

[Yoyobuae]

Because with all the suggestions of "taking out residuals" almost nobody ever thought about how inefficient the throughput gets if the assembler always stalls waiting for removal of items. It becomes way too ugly that the machine spends more time waiting for item removal/insertion than it actually does pure crafting... It becomes so inefficient that it is not worth going for in comparison to dedicating machines to specific recipes.

So my above suggestion of pipelining the input/crafting/output is absolutely necessary to increase the efficiency of the entire "Set recipes"/"Smart Assembly" concept.

Depends on the recipe really and how crafting sequence is implemented. The costs of switching recipes can be compensated simply by crafting more stuff per each cycle.

From my tests, a perfect ratio green science setup makes 4.77/minute per assembler (148/min from 31 blue assemblers) while my programmable green science assembler setup makes 4.34/minute per assembler (34.7/min from 8 blue assemblers). So 90% efficiency.

@Yoyobuae
Yes, it is a constant, however the problem with using a combinator is that a single item may be crafted in different ways. For example light oil can be either cracked from heavy oil or produced from refineries. I suppose one could limit it to assembling machines as in that case it would work.

Additional signals are needed for cases when the recipe to make something is ambiguous (see how crafting combinator mod handles it).

But anyway, your example isn't really a problem because cracking happens in chem plant and refining happens in refinery. No way recipe could be changed from cracking to refining on same building. It is a problem for solid fuel though.

But If there's additional signals for those ambiguous recipes then problem goes away.

[MeduSalem]

Depends on the recipe really and how crafting sequence is implemented. The costs of switching recipes can be compensated simply by crafting more stuff per each cycle.

From my tests, a perfect ratio green science setup makes 4.77/minute per assembler (148/min from 31 blue assemblers) while my programmable green science assembler setup makes 4.34/minute per assembler (34.7/min from 8 blue assemblers). So 90% efficiency.

Well, okay... if you put a ridiculous amount of Inserters to input the required resources and output all the residuals in "one go" then you might be able to ramp up the efficiency since you cut back on the time it takes to input/output items... but that would eventually turn out in layout problems, especially if you want to squeeze the stuff between Beacons for example.

And if I can't do the later, I wouldn't use it.

That said yeah, there also the Stack Inserters... which might alleviate the problem for Bot-based stuff. But not for belt-based since there the mixed-belt would start imposing throughput problems.

Or like voxel wrote... don't switch the recipe too often... but that means that you change to bulk crafting and with delayed response to the actual factory demand, which some people may find weird, but works nonetheless. I think the later would also require ridiculous controllers to prevent from switching recipes too early... and somehow I hate doing that kind of stuff because the Circuit Network, honestly, is a piece of oversimplified crap when it comes to programming complex processes. Makes me go into a delirious state wanting to open Notepad++ and write the entire thing as a script instead.

[Yoyobuae]

@MeduSalem: Well, I think you'll need to make some kind of compromise. Your list of requirements so far for smart assembly:

• Be belt base and botless
• Be able to switch recipes all the time
• Fit between and keep up with strips of speed beacons
• With very little use of circuit network

I don't see how that would be possible. You would at least need some kind of control circuit for mixed input belt. You can't really fit 3 input belts if it has to fit between beacon strips. And then you also need to put the output and residue items somewhere (ie. more belts).

I personally have given up making a programmable assembler setup which uses beacons. Or maybe I'll just make myself a mod which allows beacons to work a box as well.

The only way I see what you want to work is for the assembler itself to handle everything automagically. But what's the fun in that (well, maybe the devs will have fun coding it).

[MeduSalem]

Pretty much yes. Except that I'm actually playing more with bots. That said I never forget about players who might want to do things without bots.


But if I really consider how problematic recipe switching in assemblers really is then I think that the Devs will never even consider implementing it in Vanilla... because they rarely implement anything that only 1% of the playerbase is able to use properly... which would be pretty much the case when it requires an awful lot of combinator contraptions to handle everything the assembler itself doesn't.


That said I could make a rough mockup of how I think that it might actually "work" with being able to switch the recipe at any given point without any performance loss within a certain "window".

[edit]

Here it is:

Assembler Flow.png (17.87 KiB) Viewed 10157 times

Basically like #2 option of voxel could work.

1. So basically you set the recipe by signal, which then tells which items are "allowed" to be input into the internal Resource Cache.
2. Once all required resource items are inside the Resource Cache the recipe starts crafting.
3. Finished items are moved to the Finished Item Output buffer. An Inserter can pick the stuff up from here.
4. If Recipe Switch happens then the requirements of the new recipe are compared with what is already inside the cache.
5. If it turns out that the resources for the new recipe don't fully fit into the cache (= "Cache Overflow", since there may still be other mismatching items) then it proceeds to dump one mismatching resource item after another to the residual Item Output buffer until there is enough space. It could begin by dumping the "oldest" resource that hasn't been used recently to the youngest.
6. From the Residual Output buffer the inserters can grab the items that aren't needed anymore.

So basically the assembler could go on with crafting the new recipe (given that the resource items are already in place in the cache) without waiting for the residual items to be removed first. It would only start dumping residuals if there's not enough space in the cache, so it will not happen with every resource item on every recipe switch (except if all 6 resource items of the new and old recipe mismatch). The only time when the assembler would stall is either one of the following three cases:

1. Output buffer for finished items is full.
2. Cache is empty because you didn't deliver enough resources to the assembler.
3. Output buffer for residuals is full while a recipe switch requires dumping additional residuals since the Resource Cache is also full.

There would still be a circuit signal necessary to tell the output inserters that there are resdiuals or finished items for pickup. Don't know if a single signal outputting either the finished items or the residual items would be sufficient, but at least one is absolutely necessary to tell the residuals from the finished items apart. Maybe both could be provided: Finished Items would have a positive circuit signal, Residual items have a negative signal. Then you could use "Each > 0" for finished items and "Each < 0" for residual items... that way one would be able to even have several finished items for output (important for mods maybe).

The amount of Cache/Buffer Slots is up for debate of course, but I'd say the 6 slot variant is a solid foundation to work with.


So how I see it, it would be possible to store the resources for up to 6 "single-resource" recipes in the cache and switch between those 6 recipes before it would become necessary to dump something into the residual item output buffer. That in turn increases the uptime of the assembler significantly. Maybe a stall doesn't even happen at all because you get a lot of time to remove the residuals.

That in turn makes it possible to use it even with belts that may have huge delays in item delivery/export, though maybe still not as effective as bots would be of course.

So I really put a lot of thought into this... since I'm basically fleshing out the idea for almost 2 years now.