Roboports should output missing materials to circuit network / Roboports emit signals with missing blueprint items

[<NO_NAME>]

Say you build a new mining outpost. You build the train line, add the train stop, add the roboport and release some construction bots. Now you blueprint the electric miners, power poles, belts, walls, gates, flame turrets, laser turrets as suitable for the ore deposit in question. Every deposit will be different.

Assuming the roboport would output the number of ghosts the LTN train stop would then be wired to automatically request the right amount of electric miners, transport belts, power poles, walls, turrets and whatever else is ghosted. The train delivers it and the construction bots build it.

Okay.
Why do that?
I don't know about you but anything other than turrets or walls being destroyed is the exception, not the rule so in that regard, everything is the same. If you need something other than repair packs, turrets, walls and poles, that's an extremely rare emergency in which case your bot network is probably already destroyed anyway.

Why would you build a complex system that somehow keeps track of each individual outpost and somehow sends a train loaded with just what is needed there if construction is just a one time job? Like, what is there to maintain once the outpost is built?
Seems like you want to do something that just doesn't make sense.

Also if your outpost doesn't have the thing it needs on stock right where it is needed right now and it takes 10 minutes for a train to meander there, that's 10 minutes your outpost is vulnerable and once the train arrives, you'll probably no longer need precisely 2 turrets now but 10.

Keeping a few turrets in reserve at every outpost might be a good idea. But that's beside the point. You are free to stockpile.

The point was that the outpost requests the right amount of stuff to build itself without ending up with 50 extra electric miners or 1000 transport belts without the user having to specify the numbers. automation. And in case the outpost is partially overrun it will request new miners to repair itself. No need to keep a stock of 20 electric miner at every outpost just for the unlikely case the aliens break through in one place.

This!

Also, designing a fully automated system would be fun.

[dood]

Welcome to Factorio, where we always need more production, more pollution and more complexity.

Honestly, why would we NOT want to fully automate base expansion? Factorio is all about automating everything.

Okay so this is just complexity for the sake of complexity then?

Keeping a few turrets in reserve at every outpost might be a good idea. But that's beside the point. You are free to stockpile.

The point was that the outpost requests the right amount of stuff to build itself without ending up with 50 extra electric miners or 1000 transport belts without the user having to specify the numbers. automation. And in case the outpost is partially overrun it will request new miners to repair itself. No need to keep a stock of 20 electric miner at every outpost just for the unlikely case the aliens break through in one place.

There's a lot of wishy washy taking for granted that things somehow just happen going on here.
Never mind loading the thing, how would you even make the train schedule? How would you tell it where to go?
You have 20 outposts, want to make outpost #21. What do?

All I do if I build a big thing like a smelter is construct it in base, deconstruct it into storage chests, slap all that stuff into my building train and then toss it into a storage chest at the outpost so it builds itself and that's that.
What would you do in a world in which you can have roboports say "there's 1000 smelters and 2000 belts needed"?

[mrvn]

Welcome to Factorio, where we always need more production, more pollution and more complexity.

Honestly, why would we NOT want to fully automate base expansion? Factorio is all about automating everything.

Okay so this is just complexity for the sake of complexity then?

Keeping a few turrets in reserve at every outpost might be a good idea. But that's beside the point. You are free to stockpile.

The point was that the outpost requests the right amount of stuff to build itself without ending up with 50 extra electric miners or 1000 transport belts without the user having to specify the numbers. automation. And in case the outpost is partially overrun it will request new miners to repair itself. No need to keep a stock of 20 electric miner at every outpost just for the unlikely case the aliens break through in one place.

There's a lot of wishy washy taking for granted that things somehow just happen going on here.
Never mind loading the thing, how would you even make the train schedule? How would you tell it where to go?
You have 20 outposts, want to make outpost #21. What do?

All I do if I build a big thing like a smelter is construct it in base, deconstruct it into storage chests, slap all that stuff into my building train and then toss it into a storage chest at the outpost so it builds itself and that's that.
What would you do in a world in which you can have roboports say "there's 1000 smelters and 2000 belts needed"?

Welcome to the Logistic Train Network mod. Simplified you tell the stop what it needs to get and LTN will send trains to fetch the stuff and deliver it till everything is there.

[SyncViews]

What would you do in a world in which you can have roboports say "there's 1000 smelters and 2000 belts needed"?

Why I said there are other problems to solve. If you can account for "inflight" construction and logistics bots (I don't think you can right now), then you can either.

• Chain adjacent logistic networks. A network uses a circuit to tell the next what is needed, and sets that on a requester chest. Which then also requests from the next network and so on. Not sure this is 100% perfect right now, my only attempt orders too many items because logistics bots flying to the requester chest are not counted in "Read logistics network contents", so then it orders more items, and you get multiple times too many.
• Add the items to requester chests to load a train. The train needs to have all the stops pre-configured, but you can use circuits to disable the stops with no requests (so they could even share a name), and make sure each stop only unloads its "own" stuff. Should work for a single train, multiple I don't think so because it is hard to track what the underway trains are doing but one train with say a dozen wagons is already a massive amount of supplies.
  
  At the very least you could have trains with fixed filtered wagons of "everything" and control which stops are enabled and what gets unloaded. This means that the stops themselves don't need large buffers, only the train.

That functionality already exists, you can connect inserters to the network and have them deactivate if an item is above X, if you build something that number is subtracted so they shovel more of it into the network and there you go, what more do you want?

You have to store a buffer of at least 1 of the item for that to work. If the network doesn't have any, the construction bots just do nothing, you don't get for example a negative logistic signal that you could then invert for a requester chest on that inserter. If the count is low getting logistic bot deliveries one bot at a time is very slow, so for things like belts, power poles, miners, etc. you want 100's at each location.

That is a pretty large number of items, and with mods can be far greater.

Okay.
Why do that?
I don't know about you but anything other than turrets or walls being destroyed is the exception, not the rule so in that regard, everything is the same. If you need something other than repair packs, turrets, walls and poles, that's an extremely rare emergency in which case your bot network is probably already destroyed anyway.

Depending on play style and settings, miners can get overrun from time to time, and sometimes the biters take out powerpoles, belts, inserters, etc. for the turrets themselves. They also really like killing the bots.

You can make the entire bot network self-building fairly easily (the construction range is such you can build a roboport with a 1 tile logistics gap, along with a way to seed it with bots), but requires having all the items available. You can even build entire outposts/extensions remotely, but then you need a lot of items stored.

[mrzorn]

Why would you build a complex system that somehow keeps track of each individual outpost and somehow sends a train loaded with just what is needed there if construction is just a one time job? Like, what is there to maintain once the outpost is built?
Seems like you want to do something that just doesn't make sense.

Its not like building a new outpost is a thing which rarely happens ^^

I developed a trainsystem for myself where there is a supplier station at my main (named "#Supply") and requester station (all named "#Requester") at the outposts. no matter what kind of outpost, there is the same requester station turned on via a local ciruit if there are any local requests. These requests are also delivered one way via network along the rail-tracks to the supplier-station which loads a train with these items and sends the train off to the next "active" requester station. Currently I have to adjust every local demand via a constant combinator which results to always have a local stockpile of said items - if there would be a signal of requested, but not supplied items from the roboport, it could order on demand without any stockpiling.

Why?
Ask BMW, their supplychain is based on this concept. Minimal local storage, "just in time" delivery of parts.

Why in factorio? because i enjoy it that way and its a game about automation. i mean come on ^^

[ssilk]

I don't see what this would bring to the table other than overcomplication.

If you want to build really big things you need hundreds or thousands of items. And 30 or more different types of items. To make this build effectively fast (within minutes, not hours) you need so much items, that you cannot hold them all in a local store, which is refilled, when empty. Instead you need to order it, when you need it, especially, cause you cannot foresee, what types of items are needed next.

This is like real life, where you have those two storage strategies:
A) have some always used items in store. Especially when it is important. Like repair packs, walls, lasers...
B) order items in the moment when you need it. That is especially useful, if the item is expensive (roboport for example) and not much need to hurry with transport.

In practice a mix of both strategies is most useful - of course. For example have 1 or 2 roboports in storage and the rest is ordered.

The same is also valid if you want to refill your inventory: it makes no sense to have a storage of 200 destructors in all outposts. Much too expensive, but currently that’s the only strategy Factorio enables.

[ssilk]

But in my opinion both have the disadvantage, that they actively scan the areas. I also thought a lot about this and the "right" concept would be to have a list of construction for every construction network. If a ghost is placed it adds an item to that list. If placed it is removed. That's basically all, that's needed. A full scan of the area is never needed - and if then it should scan only one chunk per tick or so.

Sadly full scans ARE needed. Currently all mods can do is iterate through each logistics cell to get that cells construction area use that to find ghosts and merge the ghosts into one list.
As you guessed this is excruciatingly slow, but there is no other way since we lack any form of data and event to handle states of ghosts.
1) logistic networks can split and merge when roboports loose/regain power without ever firing an event
2) reviving a ghost or simply cancelling it does not fire events
3) caching lists of ghosts per cell is impossible due to 1 and 2, even spreading ghost scanning over multiple ticks is problematic since between ticks 1 or 2 might happen

The game should have a list of ghosts per logistic cell and expose those to the API or at least fire events for logistics and ghost changes.

Ah, that’s what I’ve been missing. But nonetheless the current algorithm of fully scanning could be optimized a lot. For example it makes no sense to scan areas far apart from player (if no automatic blueprint deployer mod is installed. Or it makes sense to scan those areas where lots of construction bots fly around. And so on.

[dood]

If you want to build really big things you need hundreds or thousands of items. And 30 or more different types of items. To make this build effectively fast (within minutes, not hours) you need so much items, that you cannot hold them all in a local store, which is refilled, when empty. Instead you need to order it, when you need it, especially, cause you cannot foresee, what types of items are needed next.

Christ, what kinds of outposts are y'all whackos building?

[Optera]

If you want to build really big things you need hundreds or thousands of items. And 30 or more different types of items. To make this build effectively fast (within minutes, not hours) you need so much items, that you cannot hold them all in a local store, which is refilled, when empty. Instead you need to order it, when you need it, especially, cause you cannot foresee, what types of items are needed next.

Christ, what kinds of outposts are y'all whackos building?

Each of my mining outposts has at least 4 compressed blue belts of ore. The rest of my base is decentralized modules joined by LTN. Together they make 2.1k SPM.
When I add another furnace module I need a few hundred productivity and speed modules. Which I certainly don't have on stock.

[dood]

Each of my mining outposts has at least 4 compressed blue belts of ore. The rest of my base is decentralized modules joined by LTN. Together they make 2.1k SPM.
When I add another furnace module I need a few hundred productivity and speed modules. Which I certainly don't have on stock.

Sounds like you don't even smelt on-site.
What giant facilities are you building per outpost?
We were talking hours with what you can do right now in-game here. Somehow.
And that roboports outputting ghosts would reduce that to minutes. Somehow.

I actually build a 2 blue belt smelter at every ore outpost which is not small and I get by just fine with a build train which I use to go everywhere so I actually have a few hundred modules as well as inserters and everything else I'll ever need on me at all times and if that's not enough, I just add another wagon and fill that with the contents of a deconstructed blueprint.

The biggest thing I built recently was an 8 belt green circuits factory and I don't exactly go around stamping those all over the map.

[SyncViews]

Does anyone have a proof of concept map that does work (can just use infinity-chest at the source network or whatever) with only such a circuit output mod/mock?

Ah, that’s what I’ve been missing. But nonetheless the current algorithm of fully scanning could be optimized a lot. For example it makes no sense to scan areas far apart from player (if no automatic blueprint deployer mod is installed. Or it makes sense to scan those areas where lots of construction bots fly around. And so on.

To an extent. You need the position the player is viewing in the map, not the character and still account for ghosts on entities being killed, etc. But there are events for all that anyway, its what happens after the ghost is placed that I am struggling with.


I looked at doing it with events only but unless can solve the other things not sure could get it working. Either a highly sub-optimal solution that I don't think is worth it, or needing a significant amount of targetted scanning anyway.

Some new lua API's could go a long way. Effectively I believe what a mod needs to know for each LuaLogisticNetwork the list of construction ghosts, deconstruction, upgrades, and item proxies. Including which already have construction bots headed to them, and events for any changes.

Keeping track of ghosts and their construction networks seems very difficult right now.

• on_built_entity is raised when a player places a single ghost (entity-ghost), so this one is easy.
• on_pre_ghost_deconstructed is raised if a ghost is cancelled by the deconstruction planner.
• on_built_entity is also raised when a player places the actual entity over the ghost, but there appears to be no way to know there was a ghost.
• Tiles are similar, on_built_entity (tile-ghost) and on_player_built_tile.
• Paste generates an on_built_entity for each separate entity (need to be careful here to keep this fast).
• Same for placing a blueprint.
• on_entity_died when an entity is destroyed, so will have to try and find if an entity-ghost was created.
• on_robot_built_entity and on_robot_built_tile when a construction robot places something. Again the ghost entity is not provided, can assume it is either an upgrade or a ghost for the new entity type.
• on_marked_for_upgrade, on_cancelled_upgrade for the upgrade planner, indicating the entity.
• find_logistic_networks_by_construction_area(position, force) will find the logistics areas, but consideration must be made for entities larger than one tile. Probably each corner point needs to be checked. And then still the decision on how exactly to output the signals for entities in multiple networks.
• on_built_entity/on_robot_built_entity and on_entity_died for roboport entities can probably be used to keep track of changes to the LuaLogisticNetwork's. By keeping track of the known state of all networks to see if one was just created/destroyed/merged/split and which ghosts were affected by that change.
• Roboports, entity-ghost, tile-ghost have unit_number to efficiently track as table keys.
• item-request-proxy does not, I think only one ever exists per entity, so can probably use the target entity as a table key.

Other things I couldn't find at all.

• Any event to know when a construction bot is headed to complete a task. Any items needed are already subtracted from the logistic network counters, and it also needs to be subtracted from any needed ghost items counters.
• When requesting modules, an "item-request-proxy" entity is created, but there appears to be no events for its creation, completion or cancellation.
• Something in LuaLogisticNetwork that can make a good table key to store required information about the network efficiently.

The first one of those 3 is the one I have a big problem with trying to stick with events. On a given tick construction bots won't actually pickup many items or build many entities, but trying to keep track of them in on_tick is far from ideal.

[Optera]

You forgot events to handle when logistic networks disconnect after roboports ran out of power and ofc connect when they are powered back up.

Instead of monitoring dozens of events just to keep track in a mod it'd be far easier to have the game keep a list of ghosts for each logistics cell.

[SyncViews]

You forgot events to handle when logistic networks disconnect after roboports ran out of power and ofc connect when they are powered back up.

Yep, that is true, and I think that is another thing that has no events.... So looks like for roboports on built/died is not enough, it may be that every roboport has to be checked to see if the assigned LuaLogisticNetwork object changed and some complex code to avoid checking every ghost entity.

Instead of monitoring dozens of events just to keep track in a mod it'd be far easier to have the game keep a list of ghosts for each logistics cell.

Its a pretty big mod interface request to have the entire thing work.

[mrvn]

I would be fine if the mod ignores power failures. But what about powering up the roboport for the first time? When roboports get build by bots they often have no power and don't connect to anything. Only when the bots also place enough power poles does the logistic network form.

How do you see if 2 roboports are in the same logistic network anyway? Does the roboport have a logistic-network-id or something? If so the mod could check for changes in the ID scanning one roboport a tick or so.

[SyncViews]

Roboports do come with some power, so actually that is another case of them loosing power.

[mrvn]

Roboports do come with some power, so actually that is another case of them loosing power.

Sure. But that only helps if the new roboport is already in range of the network it should end up in.

[ssilk]

I thought a while about that: makes no sense to count ghost-items under non-covered networks. That would end in chaotic results.

Which means, that it doesn’t matter there are ghost-roboports that are connected with an existing network. Think for this example:

One roboport on the left, another on the right. Not connected. Now you place a ghost in between, which connects both. Which is the right behavior in that case? The only correct answer is, that this can only be said if the roboport is placed and powered.

[SyncViews]

I thought a while about that: makes no sense to count ghost-items under non-covered networks. That would end in chaotic results.

Which means, that it doesn’t matter there are ghost-roboports that are connected with an existing network. Think for this example:

One roboport on the left, another on the right. Not connected. Now you place a ghost in between, which connects both. Which is the right behavior in that case? The only correct answer is, that this can only be said if the roboport is placed and powered.

I wasn't worried about ghost roboports. There should be a build event when they are actually placed, so nothing special. But ones connecting/disconnecting due to power and creating/destroying LuaLogisticNetwork objects would be a problem if there isn't an event. The whole thing becomes very complicated, and if it "desyncs" you will get strange circuit outputs.

[SyncViews]

Okay, so here is my thoughts on implementation. Not tested / written the actual code, as some important things are not complete and don't work (at bottom).

Global Data

• Need extra data for each LuaLogisticNetwork object. Factorio doesn't let me simply add extra properties, so will need to be a seperate map of some sorts to a "network_info" table. It also doesn't seem to have any unique ID, so probably an array and linear search.
  • The LuaLogisticNetwork.
  • The set of "ghost output" constant combinators for this network. These will have their signals set to all the ghosts.
  • The set of ghost entities in this network (LuaEntity of entity-ghost and entity-tile, also upgrades and item proxies).
  • A map of item->count for the items needed to build all the ghosts.
  • The set of roboports in this network.
• A mapping of each ghost to the set of network_info it is assigned to.
• A mapping of each roboport to its network_info.

Roboport Added

Code: Select all

If there is not already a network_info for this LuaLogisticNetwork then
    If this roboport has any neighbours then
        Merge them into a new network_info for this LuaLogisticNetwork
        NOTE: Merge must check for duplicate entities!
    Else
        Create a new empty network_info for this LuaLogisticNetwork
    End
End
Add any output combinators in this roboports range to the network_info set
Add any ghost entities and their items in this roboports range to the network_info set

Roboport Removed

Code: Select all

Find network_info for this roboport in global map
Remove roboport from network_info
# Handle network split
For each roboport in network_info do
  If LuaLogisticNetwork has changed
    Run "Roboport Added" for this roboport.
  End
End
# Remove entities that were connected only by this port
For each ghost in network_info do
  If ghost is not in LuaLogisticNetwork anymore then
    Remove ghost from network_info and it's item count.
  End
End
For each output in network_info do
  Remove if no longer in network
End

Ghost, Upgrade, or Item Proxy Added

Code: Select all

For each unique LuaLogisticNetwork for the entities 4 corners do
  Find network_info for LuaLogisticNetwork
  If can't find network_info, the state is broken...
  Add the ghost and the item to build it to the network_info
End

Ghost, Upgrade, or Item Proxy Removed

Code: Select all

For each assigned network_info do
  Remove the ghost and the item to build it from the network_info
End

On Tick

Code: Select all

# Could keep a dirty list, to avoid looping through every network
For each network_info do
  For each output in network_info do
    Set the output signals to the networks requested items
  End
End

Problems

If a roboport runs out of power, it is the same as "Roboport removed", but I don't see any events for it, causing state de-sync.

If a roboport then powers on, it is the same as "Roboport added", but I don't see any events for it, causing state de-sync.

Some entities might have multiple items that can place/create them. I can't really think of a way to handle this, for simplicitly, maybe always take the first item.

I still really think it is counter productive to include items that construction bots are already carrying. But there is not an event for this. Would probably have to check every construction bot (LuaLogisticNetwork.construction_robots), ideally on every tick to see if its state changed... I am not sure if there is even a way to see if a bot is active or idle without explicitly checking it.

Ghosts that are covered by multiple networks, will be reported by the outputs for each, causing a duplication. This is almost certainly going to cause problems with making use of the information in such networks.

[ssilk]

Including items that roboports are already carrying: I think that should be optional. In most cases this is useful, but think of items carried by construction robots could also mean, that they currently destruct something.

See this post: viewtopic.php?f=6&t=39474#p251107