Factorio and visual programming

[blazespinnaker]

Ok, please suspend your disbelief for a moment and please consider, especially if you're a serious programmer / datascience / ops person. I feel like this is the only crowd (factorio mod crowd) that may understand what I'm trying to achieve.

I have been playing too much factorio lately and now would like to utilize something analogical in the sense that I can compose software, deploy it, monitor, refactor, create blueprints, zoom in/out.

In particular, for data science applications. Think , Azure ML, Quartz Composer, Rabbix Composer, Yahoo Pipes, tensorflow dashboards. Really great realtime monitoring (dataflow, validation results, etc). In particular, want something that can be very definable / extensible. Eg, I can define atomic operators of the visual language, hook them up to programs/notebooks I write. The key is something nimble but very powerful.

Deconstructing, refactoring large groupings into pure code as we optimize is key to making this work as obviously this form will cause a lot of technical debt.

Larger, scratch like structural constructions are interesting as well (loop, if, etc, basic operators you'd find in a flowchart). Deployment to kubernetes would be cool, with scaling/load balancing/etc managed.

Again, the critical vision is something nimble and highly extensible, facilitating rapid refactoring and prototyping before optimization. It helps visually organizes and evokes the creative side of the mind but doesn't dumb down or cause any friction for the necessary and obviously frequent deep dive into coding internals.

It boggles my mind that something like this doesn't exist for jupyter/deep learning given the ETL / data pipelines that we work with.

I briefly considered writing a mod for factorio and using that, but there are likely too many unknowns I'd have to leap over (async REST integration, parameter adding, etc). I am now going to hack around with a react digraph editor and see if I can make that work.

But I'd much much rather use something already built, which I believe by all rights should be.

Any tips on where to look or just general feedback on the idea would be great!

[Impatient]

edit: I removed this post because I misunderstood the op's intentions.

[blazespinnaker]

I found this, but it’s unfortunately too entangled with nifi and specific use cases. I am looking for something more nimble, lightweight and extensible, ideally suited for arbitrary languages.

https://nifi.apache.org/docs/nifi-docs/ ... guide.html

[blazespinnaker]

I settled on cytoscape.js So far, very very awesome. Can't wait until I have realtime monitoring hooked up. Will post github soon. Idea is to incorporate as many ideas as possible from Factorio as it makes sense.

[Creidhne]

I might have missed something, but I don't understand what you're trying to achieve or how it's related to Factorio (and the 1st answer to this thread seems to indicate I'm not the only one).


What are you trying to deliver ?

• A standalone mod working out of the box (just download from the game portal & restart the game and it'll work).
• A 3rd party tool running outside of the game.
• A mod able to communicate with a 3rd party tool/library.

That must be answered before considering any tool. A 100% standalone mod must be written in pure LUA, without importing native code: I think it eliminates pretty much all technologies you mentioned so far. Option 3 is possible under some constraints (see Clusterio), but much more tedious to setup/launch for the Factorio player (or Factorio server admin).


What are the functionalities that it will provide to the user ?

In layman's terms and without any implementation detail ? Examples:

• My tool allows to describe a Factorio's production chain as a graph, and is able to generate a blueprint implementing this production chain (said in another way, Kirk's calculator with a graph editor and a blueprint generator). All that's left is to link the input/output belts/pipes at the edge of the blueprint and it'll work.
• My tool allows to select a part of your factory, and tell you in a graph format what it does (output example: what you selected takes up to 50 crude oil/s, and outputs X lubricant/s and Y petroleum gas/s. N refineries running "advanced oil processing" feed M chemical plants running recipe A...)
• I can compose software, deploy it, monitor, refactor, create blueprints, zoom in/out. I can define atomic operators of the visual language, hook them up to programs/notebooks I write. The key is something nimble but very powerful. Deployment to kubernetes would be cool, with scaling/load balancing/etc managed. Again, the critical vision is something nimble and highly extensible, facilitating rapid refactoring and prototyping before optimization. It helps visually organizes and evokes the creative side of the mind but doesn't dumb down or cause any friction for the necessary and obviously frequent deep dive into coding internals.

Again it might just be me missing something, but I fail to understand what the 3rd one does for a Factorio player.

[Impatient]

I read up on nifi and flow-based programming.
https://en.m.wikipedia.org/wiki/Flow-based_programming
My understanding on the op's intention is now, that how a factory works, reminds the op very much of flow-based programming. eg assemblers getting fed with inputs and once the requirements on the input side are met, an assembler creates an output which then is routed to other assemblers.

The op now wants a visual editor for flow-based programming, where the information presentation and the creation and editing actions/tools to create a flow-based program, resemble those in factorio.

The program created with such an editor, does not have to be related to factorio at all.

[blazespinnaker]

Yeah, I knew this was going to be a huge digression: "Ok, please suspend your disbelief for a moment and please consider, especially if you're a serious programmer / datascience / ops person. I feel like this is the only crowd (factorio mod crowd) that may understand what I'm trying to achieve."

When I posted this, I had been seriously considering doing it as a factorio mod and wanted to get a sense from the crowd if that made sense. Obviously, as I've learned more about what I hope to achieve, it makes zero sense. Finding cytoscape was like striking gold, as well. Can't imagine how hard it would have to build this by myself if such a framework didn't already exist.

I've posted in a lot of places and I have been struggling hard to find people to get the idea as it's a bit of a leap. Here's a really great article though: https://www.codingblocks.net/programmin ... -factorio/

Still believe factorio modding folks are my best bet as they both play factorio and program. They can appreciate the at scale / complexity UI ideas factorio has created to solve similar problems I'm looking at and are also somewhat technical so would understand the problems themselves. But Impatient seems to be the only person that sort of gets it, so maybe not.

I will report however, that I've built the early prototype of what I'm looking for and it's glorious. It has all the scale and much of the visual and adhoc capability of factorio that I think is much more aligned with at least how I perceive and work with problems at great scale. It is enabling me to solve a problem that once had me overwhelmed by the sheer scale and complexity of it, and I find a very significant decrease in context switching I often had to do with file based approaches because of the single 'map' zoom out/zoom in that I've built. This way I can stay in the flow and time seems to slip by in the same way it does when playing factorio.

Most importantly, it really helps me to maintain the mental model of the entire problem in my head at all times. As Paul Graham wrote once, this is very important - http://www.paulgraham.com/head.html

As impatient mentioned, there are definitely some dataflow / belts / inserter / assembler aspects to all of this as well that I hope to achieve. From cloud / operations point of view, there are a lot of great monitoring benefits to this. Visual Programming itself I think is very badly maligned as well, and I hope to add many more ideas behind that. Like turning a blank piece of paper on its side and drawing UML diagrams, VPL inspires and engages our creative mind as we tackle technical programs.

And there is absolutely no reason why VPL has to be for naive programmers only. I have visual code side beside my tool and I frequently dive into internal libraries for clever optimizing and enhancing capabilities of my underlying libraries - as well as debugging of course. Encapsulating and promoting VPL constructs I've drawn on the page into internal libraries is one of the early things I've built. I work with those promoted libraries with visual code mostly as the tool does become a hindrance for low level work.

Seriously though, the factorio folks should look at this. Tool building is a great biz to be in and they are obviously capable engineers.

[Creidhne]

Just for context, I initially dropped in this thread because from my initial understanding, what you were trying to achieve could have required some graph rendering/interaction in a Factorio mod, and I happen to be working on that (for a complelety different usage though). But since you've decided to go for something running fully outside Factorio (a sound decision, in my opinion), you'll be much better served by existing frameworks.

Right now what you're trying to achieve remains extremely vague to me, but potentially interesting. I wish you good luck.

[Impatient]

Thank you for the flowers blazespinnaker. I have to admit though, until you mentioned nifi and I did my research, I had no clue what you want.

When I read about flow based programming, that paradigm was new to me. So out of curiosity and because you seem to have some experience with fbp: Can you give a specific example where you used fbp or would use fbp and why that approach made or would make sense there?

[blazespinnaker]

Well, I worked at Oracle helping build a product that used spark and DAGs. A lot of people are doing this, and nifi is a good example. Azure has had something for quite awhile. There was yahoo pipes as well, but for some reason they never got it going. AWS just open sourced something, but I doubt it'll go anywhere - https://analyticsindiamag.com/why-aws-r ... evelopers/

ETL (extract / transform / load) is a typical high value scenario for data managers looking at their work from a data flow perspective.

I am actually more interested in VPL than FBP at the moment, but I hope to get to the FBP stuff soon. I just got 6 rack servers off of ebay (via labgopher) I'm setting up right now.