My interesting findings on LuaTransportLine.insert_at...

[Mooncat]

Introduction
During the development of Creative Mode, Zeblote and I discovered that LuaTransportLine.insert_at_back(items) cannot create a compressed belt. The combination of can_insert_at(0.9) and insert_at(0.9, items) seems to be able to make the belt more compress, but then Zeblote later found that it still cannot make a 100% compressed belt. Therefore, I made some studies and experiments on the functions. And I even made my implementation on ensuring a 100% compressed belt is created.

The problem is found in Factorio 0.13. We don't know whether it occurs in 0.12, but it seems no one had reported the problem before, unless I missed the keywords again.

In case someone might misunderstand the term "compressed belt", it means a fully utilized belt with no extra space between items.


The experiments
Setup
So, here come the experiments. Before that, I have changed the icon of iron-plate into this:

The ruler is used for measuring the distance between items. The rainbow is.... just for making it more colorful. (Not very useful when the ruler is added.)

And here is the setup in game for testing belt compression:

Image

For anyone who doesn't know, the purple blocks are Item Sources from my mod. They are capable of generating 2 selected items on the outgoing belts in front of them, 1 item for each line, and ideally making the belts fully compressed. In this case, they will generate the modded iron plates.
The black blocks at the bottom are Item Voids. They are capable of removing items on the incoming belts in front of them. (Yes, they have rendering order issue.) In this case, they are used to ensure the belts are moving constantly.
And since v0.0.4 (not yet released) of my mod, they support network signals, so I can turn on all of the Item Sources at the same time when I want to.
The belts with splitters are controls. Fully compressed belts are guaranteed on them.

When the Constant Combinator is switched on:

Image

By comparing the belts without splitters and those with splitters, we can know whether compressed belt can be created with our implementation on inserting items to the belts.


Results
First, the original and wildly used implementation: insert_at_back(items)
Note: actually, I have changed something here. I removed the can_insert_at_back() after checking the documents. insert_at_back will just return false if it cannot insert the item. The checking is already done there. It is totally fine for not using can_insert_at_back. Anyway, it doesn't affect the result.

Image

It is clear that it fails to produce compressed belts. The distance between items on the Express Transport Belt is very obvious.

Then, the combination of can_insert_at(0.9) and insert_at(0.9, items):

Image

It fails to produce compressed belt on Fast Transport Belt.

As noted above, I knew that we don't really need to call can_insert_at(position), so I removed it later while developing my mod without expecting much. Surprisingly, it "seems" to be able to create compressed belts!
insert_at(0.9, items) only:

Image

I have even scaled up the image to compare them pixel by pixel. Seems perfect, right?
Then I gave it a try on the faster belts from Bob's mods:

Image

Lemme borrow something from Factorio development explained in gifs

GIF

It fails on Extremely Fast Transport Belt! No need to scale up the image.


My theory
I think I might understand why they can't create compressed belts. (Forgive me if I am wrong.) The reason can be simple: item distance is not dividable by belt speed.
Since Factorio 0.13.3, item distance on belts is changed:

Code: Select all

Increased the distance between items on the belt from 0.28 to 0.28125.

And from the lua files, we know the speeds of different belts:
Transport Belt: 0.03125 => 0.28125 % 0.03125 = 0
Fast: 0.0625 => 0.28125 % 0.0625 = 0.03125
Express: 0.09375 => 0.28125 % 0.09375 = 0
Faster (Bob): 0.125 => 0.28125 % 0.125 = 0.03125
Extremely Fast (Bob): 0.15625 => 0.28125 % 0.15625 = 0.125

Why is it important? Let me visualize it with my poor drawing skill:

Image

So, here is a transport line. The dark gray square is an item just being inserted on the line. The red dot is the center of the item.
Let's say the belt is Fast Transport Belt, so the item moves 0.0625 tiles in every tick.

4 ticks after the first item was inserted:

Image

The first item is 0.0625 * 4 = 0.25 tiles in front of the insert point, which is the length of the red line above.
0.25 < 0.28125. It is not far enough to insert the next item.

In the next tick:

Image

The first item is now 0.25 + 0.0625 = 0.3125 tiles in front of the insert point. It is far enough to insert the next item.
But 0.3125 - 0.28125 = 0.03125 > 0, which means the belt is not 100% compressed!

Though I can't explain why the APIs act differently. If the above reason is correct, the results should be the same.
Also why insert_at_back cannot create compressed Express belt despite the fact that 0.28125 is dividable by its belt speed? It should take 3 ticks to insert another item at the perfect position. Maybe there is bug in the APIs?
More importantly, how can we create compressed belt with the current API?


The surprise
Later, I tried to implement the function that can insert item as near as the previous one, by remembering where was the insert point and assuming where the item is at the current tick, with the helps of belt speed + the magic number 0.28125.
Although theoretically it should work, it didn't. I didn't know the reason this time. The only thing in my head was just "go and request the API or fire a bug report!"
But then, suddenly I had an idea after observing the undesired behavior of the faulty implementation:
What if insert_at(position, items) is capable of inserting the items behind the given position in case that position is occupied?

To verify this idea, I have changed the speed of Fast Transport Belt to 0.0001, which means slow, but not stationary. Also, I have changed 0.9 to 0.5, so it now becomes insert_at(0.5, items)
Result:

Image

The right belt is stationary, to show where is "0.5". The left one moves slowly. When the first 2 items were far enough from "0.5", another 2 items were inserted. And I captured that moment.
As you can see, the items are behind "0.5"! I was right! By my observation, insert_at(position, items) can actually shift the position by half of item distance.
This partially explains why we had different results above. But can it solve our problem for creating compress belt?

I used insert_at(1 - 0.28125 * 0.5) and ran through the experiment again and here are the results:
...
...
...
nah, the same as above. It fails on the Extremely Fast Transport Belt. (Hit the ground again)

But some of you may have already known, that the Extremely Fast Transport Belt is really extremely fast.
I meant, it is so fast that new item can be inserted on EVERY TICK if the line keeps moving.
Extremely Fast (Bob): 0.15625 => 0.15625 * 2 = 0.3125
At tick N, an item is inserted.
At tick N+1, the item is moved 0.3125 tiles away from the insert point. It is larger than 0.28125, so the next item can be inserted. But there is a gap of 0.03125 tiles.
How to deal with this? Spamming insert_at(position, items) may be able to do the trick, but the result will be multiple items being inserted at the first tick and I don't want this.
So, the only way I can think of is to debug my implementation mentioned above, because we still need to know where is the last item we inserted before. After some hair pull, it is finally working now.

Image

So satisfying.

And the code:

Code: Select all

function insert_itemstack_on_transport_line(transport_line, itemstack, belt_speed, last_item_position)
	local insert_position
	repeat
		if last_item_position then
			insert_position = last_item_position + transport_belt_item_distance -- 0.28125
			if insert_position > 1 then
				-- It is outside the belt, of course the item cannot be inserted.
				-- Wait until the last item is moved far enough.
				return last_item_position - belt_speed
			end
		else
			insert_position = 1 - transport_belt_item_distance * 0.5
		end
		-- Try to insert the item at the given position.
		if transport_line.insert_at(insert_position, itemstack) then
			-- Succeeded. Remember the item position, and maybe we can insert more items at the current tick?
			last_item_position = insert_position
		else
			-- Failed to insert the item in this tick. Just wait for the next tick.
			if last_item_position then
				if last_item_position < 1 - transport_belt_item_distance then
					-- But it doesn't make sense that we can't insert a new item even if the last item has been moved beyond item distance.
					-- The only answer is: the belt is full!
					return nil
				else
					-- The last item will be moved by belt speed.
					return last_item_position - belt_speed
				end
			else
				-- It can't even insert the item at 1. Maybe the line is stopped.
				return nil
			end
		end
	until false
end

It the belt speed is higher than 0.28125 * 0.5, then you may need to use this function for creating compressed belt. (Free to use. I will be pleased if credit is given.)
last_item_position should be saved in global. You can supply nil to it in the first tick.
I don't know whether it is the best option. Tell me if you know.


However
While I am developing v0.0.4 of my mod, improving the compatibility of item-source/item-void vs underground-belt/splitter, I found out that insert_at_back(items) sometimes is the only option! I am still studying about this. Right now, I only know that:

• if you use defines.transport_line.left_underground_line or defines.transport_line.right_underground_line, it is fine to use insert_at(position, items), but you won't see the items on the input-belt.
• but if you use defines.transport_line.left_line or defines.transport_line.right_line on underground-belts, you have to use insert_at_back(items)
• for splitters, insert_at_back(items) is the only option. Inserting the items on the line numbers without "split" will make them being inserted BEFORE the split, hence they will be split later; inserting the items on the line numbers with "split" will make them being inserted AFTER the split, so no split will be done on them.

Proof (new)
Think checking image distance with human eyes isn't a good way for testing? Need mathematical proof? We have one now!
Thanks to the blueprint provided by Anson here, whose detail is explained here, my methods have been proved to be able to make 100% compressed belts even on Bob's belts.
Feel free to try the blueprint!


Conclusion

• insert_at_back(items) cannot guarantee a compressed belt, but sometimes it is the only option. (See the newly added "However" above)
• insert_at(position, items) is capable of inserting the items behind the given position in case that position is occupied. The maximum offset can be of 0.28125 * 0.5 tiles behind that position.
• However, can_insert_at(position) doesn't have that ability. So it is recommended to avoid using it. Also performance-wise, using this together with insert_at(position, items) will make the checking to be done twice, which is bad.
• When belt speed is higher than 0.28125 * 0.5, things get tricky.
• Something is still not solved, like why the Express Transport Belt cannot be compressed?
• If the devs can provide an API for us to know the current position of the last item on a line, we will be happier. (Though they sure have more important tasks to do.)

And finally, someone may be interested just like me: what about the loaders?

Image

Image

Turns out they are using insert_at_back.

Hope you enjoyed this.

[jorgenRe]

Interesting indeed

[Mooncat]

Interesting indeed

Yes, also frustrating at the same time if you don't know this before trying to make compressed belt.

[CmdrKeen]

any belt speed with a non-integer item placement frequency will result in non-compressed belts with Insert_At()

based on one insert call per tick, you get this equation:
belt speed = item size/ (desired-ticks-between-inserts+1)

belt speed 0.28125, item should place each tick.
belt speed 0.14075, item should place every other tick,
express belt speed 0.09375, item will place every 3rd tick,
regular belt speed 0.03125, item will place every 9th tick.

fast belt speed 0.0625, item will place every 4.5 ticks.*

also, even insert_at(0.999999,item) will work-

[Mooncat]

any belt speed with a non-integer item placement frequency will result in non-compressed belts with Insert_At()

based on one insert call per tick, you get this equation:
belt speed = item size/ (desired-ticks-between-inserts+1)

belt speed 0.28125, item should place each tick.
belt speed 0.14075, item should place every other tick,
express belt speed 0.09375, item will place every 3rd tick,
regular belt speed 0.03125, item will place every 9th tick.

fast belt speed 0.0625, item will place every 4.5 ticks.*

also, even insert_at(0.999999,item) will work-

well, actually, no.

Fast Belt (red belt) speed = 0.0625
0.28125 % 0.0625 = 0.03125, so it is non-integer.
And that 0.5 ticks are meaningless simply there is simply no such thing. You can only update things in each tick.

If you have read "The surprise" part, you will be surprised by how insert_at actually works.
Unlike insert_at_back, insert_at can actually insert the item at the position slightly behind the given position.
If you try to insert item in every tick, insert_at will correct the position for you and create compressed belt.
And that's why we can create compressed Fast Belts.
I use insert_at(1 - 0.28125 * 0.5, item) is because I think the maximum correction distance of insert_at is (0.28125 * 0.5), I want to give it the largest room for correction.

[doktorstick]

I definitely don't grok insert_at for splitters.

Create a splitter where one line is completely stopped and the other is being used by the test. Don't rely on the input belt to feed; feed as fast as insert_at will let you. Each tick, try to insert on the belt from 0 to 1 in .001 increments.

Code: Select all

local tout = entity.get_transport_line (output)

for i = 0,1,.001 do
    if tout.insert_at (i, item) then
        print (tick, "inserted at", i)
        break
    end
end

The above code gives the best-case insert location. It's not usable in a mod, obviously, but for the purposes of trying to understand what's going on, it'll do.

[ Edit: Reserving for future content; making progress on the algorithm ]

[Mooncat]

Thanks for your attention to this article.

I have tried really hard to understand why you have found such result, but I have no clue. I will have to make a simple mod from your snippet and test it before I know what's going on.
But there is a line especially got my attention:

That also means inserting > 0.5 is pointless since it doesn't ever insert.

Since belt's exit is 0 and entrance is 1, inserting items at >=0.5 should be more reasonable than inserting at <0.5. And my mod works well by inserting items at 0.859375 (1 - 0.28125 * 0.5). So I really want to know why it doesn't work for you.

[doktorstick]

Since belt's exit is 0 and entrance is 1, inserting items at >=0.5 should be more reasonable than inserting at <0.5. And my mod works well by inserting items at 0.859375 (1 - 0.28125 * 0.5). So I really want to know why it doesn't work for you.

Ahh. An important datapoint I didn't indicate. The entity I'm grabbing the line on is the outbound side of a splitter.

Okay, that explains why (I think) it would never insert_at > 0.5, since that would put it back on the input side of the splitter... assuming there's special code to notice that and not allow it. I need to update my post to add that that was splitter data lest someone else get confused. My pardons.

There's still some confusion, but one bit of mischief is managed.

[Mooncat]

Ahh. An important datapoint I didn't indicate. The entity I'm grabbing the line on is the outbound side of a splitter.

Okay, that explains why (I think) it would never insert_at > 0.5, since that would put it back on the input side of the splitter... assuming there's special code to notice that and not allow it. I need to update my post to add that that was splitter data lest someone else get confused. My pardons.

There's still some confusion, but one bit of mischief is managed.

Oh, yes. Splitters as well as underground belts are special cases. When I wrote the OP, insert_at had never worked on them. Because I only wanted to insert the items as near the entrance (1) as possible, I didn't try any number <= 0.5. Then, it turned out insert_at_back worked, and I can accept the fact that we can't make compressed belts from splitters and underground belts. So I stopped further study on splitters and underground belts.