Modding - Transport belt building blocks

[Ext3h]

I'm looking for a way to create custom belt types and belt fed entities, composited from single transport line fragments.

However, the API does not provide such means yet.

Provide belt building blocks. Single edge blocks which can only connect to neighboring belts, but not to other buildings blocks placed on the same position.

• Connect to the outside like a regular belt, but only 1/4th tile movement depth.
• No auto-rotation, they are supposed to be explicitly placed by the entity they are embedded into.
• They should not connect to other building blocks on the same tile, that means entities can only be passed to other positions, but not to other belts on the same position.
• The building blocks are either ingoing our outgoing, that's an explicit choice.

Bonus round, if you want to make modders happy:

• Trigger when an item reaches the end of an ingoing belt, with the option to auto-pickup that item. No unnecessary and expensive polling from Lua on the lane state!
  This also applies to regular lanes, the option to register onto an "item has stopped" event for a specific lane would be extremely helpful.
• Trigger when the back of a belt can accept a new item, rather than polling if the belt is empty.
• No GFX or sound effects required - modders can provide for that with ease.

Technically, it's "just a regular belt", with the added condition on the restricted hand over capabilities, and the restricted entity movement, which is less(!) than what could be modeled with the collision box. But it's already sufficient to allow a whole range of new entity designs.

Visualization

belt_building_blocks.png (26.76 KiB) Viewed 6824 times

What would that be needed for?

For constructing custom entities which connect properly to the existing item belt framework, without needing to hack into belt entities on neighbouring positions, respectively without the need to embed full transport belts into your design.

Entities I currently want to construct, but which aren't possible without these building blocks:

Possible applications

belt_building_blocks_example1.png (8.42 KiB) Viewed 6824 times

Line splitter. Split two input belts into two output belts, whereby each belt is split by line.
Straight out impossible right now.
Abusing underground belts for that task surely works for the regular user, but that's not a solution for modders.

belt_building_blocks_example2.png (9.84 KiB) Viewed 6824 times

Inline 3to1 round robin merger with line balancing.
Yes, this could already be emulated right now by using a regular piece, but you can't achieve the round robin behaviour, and line balancing only works to a limited extent, due to spacing on the belts in normal operation.

belt_building_blocks_example4.png (11.47 KiB) Viewed 6820 times

Inline 1to3 splitter with line balancing.
Straight out impossible right now.

belt_building_blocks_example5.png (12.47 KiB) Viewed 6820 times

Inline 1to3 splitter with line conserving.
Straight out impossible right now in that form factor, requires at least 1x3 space with manually hacked splitters, respectively 2x3 and two regular splitters with stock parts.

belt_building_blocks_example3.png (4.16 KiB) Viewed 6824 times

Powered belt. Turn lanes on or off individually based on logistic network properties or power. This is the equivalent to the electric pump for fluid networks.
Right now, this could be emulated by placing and removing a full belt per script, but that has nasty, nasty side effects as adjacent belts are altered as well.
This also doubles as the long requested belt gate.

belt_building_blocks_example6.png (4.67 KiB) Viewed 6816 times

Belt fed high throughput factories, with proper visuals on the entry lines.
If you would use regular belts for feeding, this would unintentionally allow feeding from the sides, which could conflict with the graphic concept.

Helpful side effects

• If e.g. a factory makes use of outgoing belt fragments, inserters can still pick items from the belt stub - regardless of whether a belt is attached or not. That's most helpful for high throughput, NOT inserter based production lines, without limiting the players choices.
• Inputs and outputs being static means that neighbouring transport belts don't randomly change behaviour.
• Independent inputs and outputs can be used on the same position - without unintended side effects such as items flowing through and alike.

Related topics

• Better control of transport lines Request being able to control item flow between lanes.
• The Lane swapper ☸ Request for line swappers.
• [0.11.x] Belt Utilities Nasty hack-job abusing custom collision boxes on splitters to achieve at leas some control. No longer maintained.
• Idea Some advanced transport belt Prior request of the lane splitter functionality.
• Numerous other mods including fully fleshed, hidden transport belts into their entities, with uncounted side effects and a significant impact on space consumption. You literally can't do it compact as soon as you need more than one belt.

Consent in most cases was that it shouldn't be stock behavior, and I can't disagree on that. But modders should not be hindered from implementing such things if desired for their own projects.

[GotLag]

• [0.11.x] Belt Utilities Nasty hack-job abusing custom collision boxes on splitters to achieve at leas some control. No longer maintained.

"Nasty hack-job"? In 0.11 belts were physically simulated and so blocking outputs was a perfectly cromulent solution, with no performance impact. It's no longer maintained because it no longer works.

I'm not sure where you're going with "building blocks" but if you mean having a way of defining transport lane in/out connections, with some way of controlling which input lane each output draws from, then I'm on board.

Presumably the prototype format would be similar to fluid box connections, with a parameter for outputs that lists which inputs to cycle through. Being able to connect production inventories to lanes would be nice but loaders look like filling that niche.

[ssilk]

How about having the ability to control this via (internal) circuits?
So an item comes in on a lane and you know also which item type it has. You have then a limited time to decide on which lane/direction it should be put.

[Ext3h]

I'm not sure where you're going with "building blocks" but if you mean having a way of defining transport lane in/out connections, with some way of controlling which input lane each output draws from, then I'm on board.

Presumably the prototype format would be similar to fluid box connections, with a parameter for outputs that lists which inputs to cycle through. Being able to connect production inventories to lanes would be nice but loaders look like filling that niche.

"Cycling through" is a possible optimization that would allow simple belt-type entities to work without LUA logic at all, but the question would then be if that is really the most common use case.

I was thinking a more generic solution, where that item routing is handled by LUA code, as even trivial lane balancing, splitting or merging and alike is already beyond the capabilities of a simple "cycle through" notion. That's certainly not as fast as native routing capabilities, but for mods, "fast enough" should be sufficient.

The comparison with the loader is actually quite fitting, think of the "building blocks" as just the belt connecting part used on the loader, but with the option to hook arbitrary logic onto it. Might be an inventory, might be a factory, might be the back end of another lane.

How about having the ability to control this via (internal) circuits?
So an item comes in on a lane and you know also which item type it has. You have then a limited time to decide on which lane/direction it should be put.

That's what I meant to solve by providing "triggers" when the item hits then end of the lane (respectively the "teleporting point"), allowing to handle that logic with only a single roundtrip to LUA, rather than the expensive polling approach where the entity has to actively query for the contents of the belt piece.

A tagging based switching solution as you described does provide for smarter Splitters and alike, but it's not necessarily faster than doing it per callback, and it's limited to scenarios where you always want a "pass through" behaviour as fallback. It's not providing the option to stop the item flow all together, or to model any type of "consumer" scenario.

IMHO the most elegant solution to provide that, is allowing callbacks for "items has reached end of lane" and "back of lane can accept another item", that's sufficient to handle both a spare input belt, as well as a backlog on the output belt in an efficient manner with zero polling.



As opposed to the original post something I've become aware, is that the point where the item stops shouldn't actually be fixed at 25%, that's something which should be left configurable. So that the point where the items stop can be adapted to the visuals, up to the point, where input and output lane of the constructed entity match seamlessly. (As if there was a forward-teleportation-free splitter, except that there isn't any *actual* continuous lane, it just looks like that to the user.)

[ssilk]

What I'm quite sure about is, that this should not be opened to Lua Code. This is by far too slow.

[ratchetfreak]

Making this "acceptable" for a pass through situation and still having full control is not trivial.

There is an option for providing a table per input with a set of item filters and output (or callback) pairs.

Add the option to specify a round robin of the provided outputs and it should be good. The callback option is there so attached inventories can be handled (returning true if the item is to be removed from the belt, false if not (will not call the callback again until the item on the output is explicitly removed))

API would be something like


for inputs:

peek_next_item() returning the item on the end of the line. (or null if no item waiting)
pop_next_item() returns and removes the item on the end of the line. (or null if no item waiting)
set_auto_connection([{items:[...], outputs: [...]}, items:[...], outputs: [...]}, items:[...], outputs: [...]}]) for the filtered auto connection outputs can also accept a callback which takes a reference to the lane and the item


for output there would be just
push_next_item(...) which takes the next item to push onto the belt, errors if there is no room for it
can_push_item() to check whether you can push

[Ext3h]

What I'm quite sure about is, that this should not be opened to Lua Code. This is by far too slow.

Not even remotely sure about that.

What's the current performance level of the chosen LUA engine? Something around 2-5k LUA functions invoked from native code, per second, on a 2Ghz C2D? Proportionally more on a modern CPU?
Sure, that's still magnitudes slower than native code. But then again, the current solutions found in mods are doing it by polling(!) on blue belts. So that's a static overhead every single frame, while also just (at most) a single item is transferred. Plus the callback solution is mostly for producer/consumer situations, where logic is required anyway.


@ratchetfreak
Thank you, looks great.

Your draft for the set_auto_connection() function looks reasonable.

I think setting up auto connections like that should actually work to handle all pass through cases efficiently.
Registering EITHER a list of outputs (for automatic handling) OR a single callback (for custom logic) per filter sounds good. Mixing callbacks and outputs for different filters provides the flexibility required.

The same item type should be allowed to show up in more than one filter clause. The first matching clause which ALSO has an output / callback which does accept the item, consumes it. (Provides priorities.)
If an output list contains more than one lane, the outputs in that list are cycled through for every item accepted. (Provides round robin.)
The filter list should be evaluated right away on being changed. (In order to trigger consumption of already waiting items.)

Some open question though:
- How to provide round robin when merging lanes? That's only solvable with a callback, I think. (Not a problem though, as that IS already beyond of what any stock Factorio component can do.)
- Whats the behavior if the item could not be accepted by any output lane nor any callback? Try again next tick? Output lanes can be monitored, that's not much of a problem. Callbacks can't be "monitored".

The latter one is really tricky. IMHO, a callback which has denied once shouldn't be called again just right away. It should be forced to re-register. It should only keep getting notified as long as it behaves stall-free.



can_push_item() could also accept an optional callback, which will be invoked if the return value was false.

Alternative implementation: Calling it "on_can_push_item(callback)" right away, and invoking the callback right away if an item can be pushed.
The callback should remain registered as long as it returns an item (to be pushed) right away, and unregistered otherwise.

Without that, stalled output belts would cause unnecessary overhead as there would need to be an onTick handler polling all the time.
In case the "auto_connection" callback was discarded for blocking (because it waited for an output lane to become free), this event would be the correct one to re-register the callback.



Having these producer/consumer type callbacks with forced de-registration ensures that the entire entity is pretty much guaranteed to be modeled as an event driven state machine, with zero risk of polling. (Unless the user jumps loops to bypass the API design.)
I expect that for each connection, there is always only either a producer, or a consumer callback registered, depending on whether output or the input is stalled. And if the entity is internally stalled (e.g. factory with production time), none is registered at all.



No objections on the other methods, they look all sane.

[ratchetfreak]

I did add the phrase "will not call the callback again until the item on the output is explicitly removed" to avoid excessive polls into the mod code. And it would put the burden onto the mod API to fix the clog.

[Ext3h]

Ah, so you meant "The callback is only called once per item". And I assume you meant to say "Input", not "Output".

Not the same as explicit removal though, an item being handled by lane-to-lane transfer later in the filter list would also be removed, just not explicitly.

And that still means a call into the mod code for each single passing item, regardless of whether the modded entity is internally clogged (or output restricted) or not.

The ideal however is to have a cost linear in the number of actually processed items, not in the number of arriving ones.
Otherwise, the pass through functionality brings no performance benefits as soon as there is any callback registered at all.



Let me try to model a few scenarios. Let's say we have a 1x1-entity with. And for simplicity, let's enumerate the lanes by cardinal direction, with the input in the south such as lane<side><lane>. I'm going to omit the explicit notion of input and output lanes.


An sieve for stones, dropping stones from each lane to the sides and routing everything else straight through, would then look like this:

Code: Select all

laneSW.set_auto_connection([
    {items: ["Stone"], outputs: [laneWN, laneWS]},
    {items: ["*"], outputs: [laneNW]}
]);
laneSE.set_auto_connection([
    {items: ["Stone"], outputs: [laneEN, laneES]},
    {items: ["*"], outputs: [laneEW]}
]);

Lane balancer:

Code: Select all

laneSW.set_auto_connection([
    {items: ["*"], outputs: [laneNW, laneNE]}
]);
laneSE.set_auto_connection([
    {items: ["*"], outputs: [laneNE, laneNW]}
]);

The belt splitter from the OP:

Code: Select all

laneEN.set_auto_connection([{items: ["*"], outputs: [laneNE]}]);
laneES.set_auto_connection([{items: ["*"], outputs: [laneSE]}]);
laneWN.set_auto_connection([{items: ["*"], outputs: [laneNW]}]);
laneWS.set_auto_connection([{items: ["*"], outputs: [laneNS]}]);

Belt-fed smelter:

Code: Select all

function takeOre(lane)
    -- This is called way to often....
    if hasOre then return false end
    item = lane.pop_next_item()
    -- Do stuff with item
    return true
end
function takeFuel(lane)
    -- This is called way to often....
    if hasFuel then return false end
    item = lane.pop_next_item()
    -- Do stuff with item
    return true
end
function smelt()

end
function smeltFinished()
    for lane in [laneNW, laneNE] do
        if lane.can_push_item() then
            lane.push_next_item(output)
            return
        end
    end
    -- This is now a problematic situation, this entity got output stalled!
end
for lane in [laneSW, laneSE] do
    laneSW.set_auto_connection([
        {items: ["Ore"], callback: takeOre},
        {items: ["Coal"], callback: takeFuel}
    ]);
end

Hrm...
If a callback happens to store a reference an item without actually returning true, then it would cause possibly invalid references.
And a callback returning true without actually destroying or storing the item results in leaking that item.


A callback when a clogged output lane becomes free is inevitable.

Well, no, actually you would want to be able to wait on multiple output lanes at once, and just be informed which the first free one was. E.g.:

Code: Select all

-- Just a draft
function on_can_push(lane_list, callback)
    for lane in lane_list do
        if lane.can_push_item() then
            callback(lane)
            return
        end
    end
    -- And now the nasty bookkeeping would begin...
    -- The hackjob solution would be to repeat this now every single game tick...
end

-- Example use
on_can_push([lane1, lane2], function (lane) lane.push_next_item(item) end)

It's essentially the same requirement as on the output lane for set_auto_connection().

I am not sure about leaving this callback permanently registered though. Respectively what the logic for "no polling" would be. You wouldn't actually want to keep this callback to be triggered on every single gap, but only if the callback could have an effect in the first place. So one-shot is more reasonable.

However, worst case scenario would actually be to splice a dense belt onto two half-clogged belts via logic with a minor backlog, as that would mean a fast toggle between input- and output stalled. And for that, you wouldn't want to re-register all the time either...

That's no good, there are legit arguments pro and contra one-shot callbacks.

Trinary return values could be the solution, "I took the item / filled the slot", "I did nothing, but keep me posted", "I'm done, sign me of". Whereby the first one can actually be turned into an explicit choice, by explicitly calling push/pop on the corresponding lane. The first callback modifying the lane would then stop the bubbling, while the return value only designates if the callback is to be re-used ASAP.

Positive side effect: That provides also the ability to build belt throughput counters, as requested multiple times.

I'm actually quite curios, how the handover from items onto stop and go belts is currently handled internally. With a bit of luck, there is already some suitable structure in place which only needs to be exposed.

[Hexicube]

The first proposed use of a belt splitter (NS to EW) is actually possible, and I've actually made one that functions. The issue is that there's no way to check where on a belt piece an item is, and therefore no way to know that the item is at the end of the belt.

[Ext3h]

The first proposed use of a belt splitter (NS to EW) is actually possible, and I've actually made one that functions.

Size 1x1, or full 2x2/3x3 with entirely separate, full input and output belts? Yes, it's possible with the latter size, and massive polling.

Actually, all of them are possible, if you treat the neighboring belts as part of the entity as well, and with polling every single simulation frame. But not only means that having a full 1 wide belt border around your entity for belt connections, but also pretty ugly scripts, ugly in terms of performance, mostly, but also in terms of visuals with teleporting items.

[Hexicube]

The first proposed use of a belt splitter (NS to EW) is actually possible, and I've actually made one that functions.

Size 1x1, or full 2x2/3x3 with entirely separate, full input and output belts? Yes, it's possible with the latter size, and massive polling.

1x1, constant polling of adjacent belt feeds to check for items. The issue is it takes from the belt the instant the item is on the belt, which looks odd. Can handle full blue belt throughput too.

[Ext3h]

1x1, constant polling of adjacent belt feeds to check for items. The issue is it takes from the belt the instant the item is on the belt, which looks odd. Can handle full blue belt throughput too.

Ah OK, so you are treating the neighboring belts as if they were part of your entity, with all the possible side effects of belts snapping with other belts in unwanted ways.

And how many of these blocks can you place before the game starts slowing down noticeably? Shouldn't be more than about a hundred in total, before the polling becomes too expensive to handle in realtime.

[ssilk]

And how many of these blocks can you place before the game starts slowing down noticeably? Shouldn't be more than about a hundred in total, before the polling becomes too expensive to handle in realtime.

That's what I meant with: Could not use Lua for that, cause it's too slow.

[Hexicube]

1x1, constant polling of adjacent belt feeds to check for items. The issue is it takes from the belt the instant the item is on the belt, which looks odd. Can handle full blue belt throughput too.

Ah OK, so you are treating the neighboring belts as if they were part of your entity, with all the possible side effects of belts snapping with other belts in unwanted ways.

And how many of these blocks can you place before the game starts slowing down noticeably? Shouldn't be more than about a hundred in total, before the polling becomes too expensive to handle in realtime.

Didn't experiment with CPU load, since I was disappointed with the results. Most of the CPU load would probably come from repeatedly checking if a belt is there and getting the rails for the nearby belts, I should really be storing if a belt is there (and the two belt rails) and use the listeners for when entities are added and removed (which I do use for my own entity).

Belts function exactly the same regardless of my entity being there, they're only checked for input if they point directly into my entity and are only considered for output if they do not. If an item exists on the belt lane I check, I ask the appropriate output lane if it can accept an item, and if it does I move the item over. My entity doesn't currently store items, it was an experiment to see if it was possible (which it is).

[ratchetfreak]

Another output option: add to inventory (or just some slots)

To allow for loader like mechanics and to allow the mod to just check whether there is enough stuff in there and then remove the items and do its thing.

Similarly: auto feeding an lane output from an inventory.

[Ext3h]

Another output option: add to inventory (or just some slots)

To allow for loader like mechanics and to allow the mod to just check whether there is enough stuff in there and then remove the items and do its thing.

Similarly: auto feeding an lane output from an inventory.

I like that, a lot. Question would be how that behaves for chests, with respect to blocked slots. Hard wired to the same rules as an inserter?

For auto feeding, that's slightly more complicated, as there is no notion to fetch from inventory yet.


Hrm, this is slowly mutating into a generic notion of item transfers... Not that I'm complaining, just saying, and also pointing out, that this has a few implications, such as that the from.declare([{filter, target}]) notion should probably be generalized to ItemRouter.(source, [{filter,target}]). Maybe even ItemRouter.([source], [{filter,target}]), and returning a reference on the route, so that it can be deactivated, or having callbacks replaced later on.