Extract Industrial IoT data to learn

[augustinus]

Dear community,

I'm a long term player of Factorio but just recently looked into modding it. I'm learning about Indistriual IoT use cases, e.g. calculating the Operational Efficiency of a factory, supply chaing optimizations, etc.

One issue I noticed was that it is hard to learn the necessary tools to collect data, visualize them, and create the calculations as it is very hard to create meaningful demonstrations and understanding the limits of various approaches. You can hook up a few sensors and create a small manufacturing line with some sensors, but it is not nearly on the right scale to get into certain limits and also quite costly and takes of a lot of space.

So my idea was to create a mod for Factorio that extracts IoT data of all the involved components. That would allow me to create a fairly complex factory line and ingest the machine data into Industrial IoT software, creating analytics on the data, calculating the efficiency and finding ways to optimize the processes.

I played around with the modding toolkit but I'm having issues to extract the data I need for it. For example: I can get a list of picker entities as a list of LuaEntity objects, but there is no indication if the picker is currently picking and how many objects have been picked (like a counter value).
I figured out that some buffer chests allow me to count the number of items at a special positon but I cannot get throughput data. There are also no events on machine level, I could register to.

Did someone work on something like this before? Am I looking at the wrong places?

Thank you for your help

[robot256]

This is an interesting topic. I'm not totally clear what all the data you are looking for is. Are you trying to log data over time while running a map in the game, so you can plot it afterwards? Can you list what exactly you are trying to log, so we can identify sources for each piece?

Some of the data you are looking for is available inside the game via the circuit network. For example, you can make a combinator circuit to measure the total or average throughput of a belt or inserter by setting the belt or inserter to "read contents / pulse mode", so that each item produces a one-tick circuit signal when it enters the belt/inserter, which is then summed/integrated by combinator memory cells. I don't know an easy way to get this via mod scripts besides creating a circuit network and reading it.

The contents of a chest or tank is trivial to read in both circuit network and mod scripts.

The LuaEntity::status field gives detailed status for inserters and assemblers that mods can read. You will have to read it on your own timer (inside on_tick or on_nth_tick) because atatus changes make no events.

[augustinus]

Thank you for this great ideas. Yes, I was thinking about ingesting this data into a tool like AWS IoT SiteWise. As machines in factorio don't have wear and tear I don't need specific data of that but my idea is:

- Raw materials are stored in buffer chests. The chests are filled, which is the delivery you get from your supply chain
- I want to create 2-3 end products, that require the raw materials to go through several fabrication machines to build all intermediate components needed for these products
- All fabrication machines, furnaces, pickers emit data every time they complete an action. This allows me to report the overall efficiency of the plant and to detect where an improvement would make most sense. As we all know: optimization has to be done on the critical path, otherwise it will have no effect.
- I'm also thinking to collect power consumption data, to create an emission reporting dashboard based on the energy mix
- I'm also thinking about randomly reducing the health of the machines every n minutes to simulate wear and tear. You can schedule planned maintenance to bring the health back up or the machine will be deactivated if it drops too low and will be down unplanned and much longer. The tools then help to find the ideal maintenance windows to keep your factory running as smooth as possible.

I'm not thinking about a huge endgame Factorio factory with tens of thousands of machines and Terrameters of belts. I'm thinking of a small factory process that fits on a screen but still is way more complex that everything you could build physically and allows you to do modifications on the fly and see the effects.

I will dig into circuits and memory cells.

Another thing I'm struggling with: If I get the devices via find_entities_filtered I don't see an attribute in LuaEntity that allows me to identify this device, like a unique ID. Or the coordinates where it is placed. I was thinking of creating my own idea, e.g. by combining the picker type, the pick-from and the insert-into coordinates, as this combination should be unique.

As my factory will be quite small I have the benefit of limiting myself to a few device types for the start.

[Qon]

Modding in a type of combinator that logs its inputs to file together with a constant ID signal (from a constant combinator or specified on the logging combinator) seems like a a good way to get the data out of Factorio without making a custom solution for logging just 1 specific Factory. An ID combinator mod could then be used together with the logging combinator mod to not have to manually input numbers inside Factorio at least. Though the ID numbers might have to be mapped to some label inside the IoT data management software you want to use if not the signal types and the positions in the log messages are enough. position could come from a positional combinator or the logging combinator could just automatically insert its own coordinates in the log messages (and then also generate its own ID maybe, but having these features external to the logging combinators gives more flexibility).

I might consider throwing together such a combinator mod if no one else does it first, I have done similar mods before.

[robot256]

It may even be possible to use a Lua combinator mod like https://mods.factorio.com/mod/Moon_Logic to call https://lua-api.factorio.com/latest/cla ... write_file to log circuit values, at least as a prototype. Qon's idea 9f having tagged value logger combinators so everything ends up in the same file would be a good solution.