Bigger more dynamic mods with less memory

[Zhog]

I apologize in advance for breaking rule 6, but there just is no short way of explaining this.

Lets assume you are writing a mod for a furnace the uses oil instead of electricity or coal. Right now there is no easy way to do this. But if objects were defined in XML instead of being hard coded there would be.

Lets assume the XML definition for a furnace looks like this

Code: Select all

<item name="stone_furnace" icon="__base__/graphics/icons/stone-furnace.png" order=1>
	<recipe>
		<ingredient type="stone" amount=5>
	</recipe>
	
	<flags>
		goes-to-quickbar
	</flags>
	<subgroup name="smelting-machine" />
	<stack size=64 />

	<place-result>
		<flags>
			placeable-neutral,
			placeable-player,
			player-creation
		</flags>
		<minable time=1 result="stone-furnace" />
		<health max=150 />
		<corpse type="medium-remnants" />
		<repair-sound filename = "__base__/sound/manual-repair-simple.wav" />
		<mined-sound filename = "__base__/sound/deconstruct-bricks.wav" />
		<resistances>
			<fire percent=80 />
			<explosion percent=30 />
		</resistances>
		<colluision-box upper-left="-0.7, -0.7" bottom-right="0.7, 0.7" />
		<selection-box upper-left="-0.8, -1" bottom-right="0.8, 1" />
		<inventory type="source" size=1 />
		<smelting speed=1 energy-consumption="180kW">
			<item name="iron-ore" amount=1 energy-required=3.5>
				<result name="iron-plate" amount=1 />
			</item>
			<item name="copper-ore" amount=1 energy-required=3.5>
				<result name="copper-plate" amount=1 />
			</item>
		</smelting>
		<energy-source type="burner" effectivity = 1>
			<inventory size=1 />
			<emission type="pollution" amount=0.01 />
		</energy-source>
		<smoke deviation="0.1, 0.1" frequency=0.5 position="0, 0" vertical-speed=0.05 />
		<on-animation filename="__base__/graphics/entity/stone-furnace/stone-furnace.png" priority=10>
			<frame width=81 height=64 count=1 shift="0.5, 0.5" />
		</on-animation>
		<off-animation filename="__base__/graphics/entity/stone-furnace/stone-furnace.png" priority=10>
			<frame width=81 height=64 count=1 shift="0.5, 0.5" />
		</off-animation>
		<fire-animation filename="__base__/graphics/entity/stone-furnace/stone-furnace-fire.png" priority=10>
			<frame width=23 height=27 count=12 shift="0.078125, 0.5234375" />
		</fire-animation>
		<fast-replace group="furnace" />
	</place-result>
	
	<control-script filename="" />
</item>

When a stone furnace is in use, the relevant parts of the XML definition would be parsed and the matching functions would be called.

Now writing an oil furnace would be very simple.
First you would need to create the fuel type.

Code: Select all

<energy-source name="fluid">
	<fuel type="oil" value="100MJ" />
</energy-source>

Then all you would need to do is go into the furnace definition change the name, possibly change the image and icon filenames, and the energy source tag from "burner" to "fluid".

Now you have a furnace that accepts oil as fuel, but what about the input? That's just as easy as adding one XML tag to the furnace definition.

Just add:

Code: Select all

<pipe-connection type="input" position="1, 1" />

Now your mod is complete with no complex or multiple lua files and minimal time, and as a bonus this mod adds only a single file pointer into memory. This allows for much larger mods with little performance impact.

[FreeER]

first, moved to modding interface since it's more appropriate there.

Second, I'm no xml expert but I believe xml is actually (far) less flexible than lua (see the implementation of shotgun ammo, data\base\prototypes\item\ammo, or the combat robot tech for instance, no idea how you'd do those with xml) and even if it was possible it wouldn't solve the problem because the C++ coded binary (aka, factorio.exe) needs to know HOW to load that xml which means hard coding the loading...and you're back to where you started. What is really needed is a more 'generic' (probably not the proper programming term but...) way of specifying implementation details (such as energy sources), but that takes proper planning (which takes a fair amount of time) and work (more time) to do, so it's been kind of at the bottom of the dev's todo list. As an extra you'd still need lua for the control file so now instead of people with no programming experience having to learn lua (for anything complex, oh and partially json for the info file...) they'd also need to learn xml, not that it's difficult but... so, no real reason to switch.

btw, as for multiple files, they CAN all be done in a data.lua (the only reason to separate them is to make maintenance and updating easier) as for the 'performance' impact lua CAN be compiled instead of purely interpreted. It's not really a major issue at this time however.

[ssilk]

I've been looking many times into this lua-code-configuration now and I think it is quite efficient, how it works.

And it is also not the only way you can create it....you can split everything into single parts, make anything you want with it. There speaks nothing against having a XML to LUA converter, which adds the needed stuff. There are some XML to LUA converter out there, i'm nearly sure.

The problem with converting this from XML into LUA (and it should be LUA in the end!) is, that the metainformation (how should this converted, for example via XSLT) needs also to be generated then. Double work.

[Zhog]

I forget sometimes that I'm not talking to my coworkers. So here comes the long winded explanation that I was trying to avoid before.

It does seem like the XML would be less flexible. You have pointed out that the main impediment to modding is C++ classes. So lets get rid of those all together. With an XML interface those would not need exist. Instead there would just be a set of hard coded type independent functions that are called. This needs a change in thinking, so instead of thinking of a class as an object, think of a class as a definition of what an object should be. I'll show you an example by converting freeER's bomber. Something to keep in mind as you read this is that I have not actually read the source code for the game, this is just my opinion of how it should work and is based on my own experiences as a professional programmer.

So lets assume that these are some of the global functions available to modders.

Code: Select all

create_group(char *name, char* parent, FILE *xml_file)
/*
Creates a group and adds it to the internal group tree structure
if there is a parent string then add this group as a child node of parent
*/

create_item(char *name, FILE *xml_file)
/*
Creates an item with the given name is stores it and the xml pointer in an internal item tree
*/

create_technology(char *name, FILE *xml_file, bool locked)
/*
Create a technology and add it to the tech tree
*/

unlock_technology(char *name, FILE *xml_file)
/*
traverses the technology tree and unlocks the tech pointed to by *name
*/

create_recipe(int *item, FILE *xml_file)
/*
Create a recipe for the item pointed to by the item pointer and set the ingredients for it by
parsing the XML
*/
 
consume_fuel(float amount, FILE *xml_file)
/* 
Consumes an amount of type independent fuel
*/ 

emission(float amount, FILE *xml_file)
/*
Emits an amount of pollution
*/

on_key_press(int key, FILE *xml_file)
/*
First this function check to see if the key pressed is a base function (like pressing "e" 
will always open the crafting screen). next the function searches through the item
tree and looks at the XML file for <on-keypress> tags. If the key is a match then
run the associated lua script.
*/ 

accelerate(float energy, float rate,  float friction, int weight)
/* 
Any item with the "pushable" flag will call this to both accelerate and decelerate.
We can get away with only one function because deceleration can be expressed
as acceleration in reverse
*/

place_item(int *item, int x, int y, FILE xml_file)
/*
Places an item, at x,y , that is described by *item
*/ 

animate(FILE *spritesheet, int frameWidth, int frameHeight)
/*
This function will be called each tick for every item in the scene graph. it will update
it's current frame and draw it to the back buffer.
*/

So lets start with the easy part and define our groups that will appear on the crafting screen.

Code: Select all

<group name="bomber" icon="__bomber__/graphics/bomber.png" order=4>
	<subgroup name="planes" order=1 \>
	<subgroup name="bombs" order=2 \>
</group>

No conversion to or knowledge of lua is needed at this point since there is no complex logic to execute. Instead, when the game is loaded it checks the XML in the mods directory. When the "group" tag is encountered the create_group function is called for each group and subgroup tag. No classes are extended and no new classes are created or inherited. This is where the memory thing comes into play. It takes less memory and processing time to parse a file then to create an instance of a group class. It's just a few fractions of a difference, but those fractions add up over time and increase with the complexity and inheritance level.

Now lets move on to something a little more complex and create a bomb.

Code: Select all

<item name="bomb" icon="__bomber__/graphics/bomb.png" subgroup="bombs" locked="true">
   <stack size = 5>
	<recipe>
		<ingredient type="electronic-circuit" amount=10 />
		<ingredient type="iron-gear-wheel" amount=10 />
		<ingredient type="iron-plate" amount=20 />
	</recipe>

	<flags>
		goes-to-main-inventory
	</flags>
        
</item>

Going step by step, the first thing that happens at load time is the XML parser encounters and "item" tag. The program then calls the create_item function which creates an item of name "bomb" the uses the icon located at "__bomber__/graphics/bomb.png" and then places a pointer in the item tree in the "bombs" node with the parent of "bomber". Notice there is no type. Because there are no hard coded classes there is no type to extend.

Next the parser encounters the "recipe" tag and calls the create_recipe function. The recipe and it's ingredients are saved in a table.

Now without using lua or even having any know of lua or the internal data structures of C++ we have just created a completely dynamic class the can be created at runtime instead of compile. It may not look like a C++ class and that's because it's not a traditional class definition. However, when you get right down to it, a class in C++ is nothing more than a highly restrictive data structure. Knowing this means if we define our own data structure, we can in essence create a class at runtime that would look something like this:

Code: Select all

/* Item Tree */

                                     (  items  )
                                  /       |       \
                                 A        B       C
                                          |
                                      (bomber)
                                    /           \
                                (bombs)    (planes)
                                 /
                           [bomb] <-- Item struct
                           locked = true
                           stack = 5
                           recipe =
                              [
                                 [*electronic-circuit, 10],
                                 [*iron-gear-wheel, 10],
                                 [*iron-plate, 20 ]
                              ]
                           enum flags = goes to main inventory

Now lets move on to something more complex: the bomber. we start off the same with the same tags.

Code: Select all

<item name="bomber" icon="__bomber__/graphics/bomber.png" subgroup="planes" locked="true">
   
   <recipe>
      <ingredient type="iron-stick" amount=50 />
      <ingredient type="electronic-circuit" amount=50 />
      <ingredient type="iron-gear-wheel" amount=50 />
      <ingredient type="iron-plate" amount=200 />
   </recipe>

   <flags>
      goes-to-quickbar
   </flags>

  <technology name="bombing" icon="__bomber__/graphics/bomber_tech.png" prerequisite="flying">
      <effects>
         <unlock recipe="bomber">
         <unlock recipe="bombs">
      </effects>
      <ingredients count=100 time=20>
			<ingredient type=science-pack-1 count=2 />
			<ingredient type=science-pack-2 count=1 />
			<ingredient type=science-pack-2 count=1 />
      </ingredients>
   </technology>
</item>

So now we have created a bomber class and just like before it's been placed in the item tree. Like the bomb, It's locked and we can't place it yet, but I'll get to that in a minute. So again, at runtime, the file is parsed. Same as the bomb, the bomber an entry is created in the item tree. Although this time there is a technology. When the parser encounters this tag it will call a create_technology function which in turn will create a new entry in the tech tree with the parent entity of "flying".

Now here we are in the game. Our new items have been created and placed in the crafting menu int their own tab. We can't see them yet because they are locked. So lets do our research the same way it's always been done. When the research is done, the internal functions of the game go back to the XML file, skips straight to the "technology" tag, and the parses only this tag looking for the "effects" tag. Inside the "effects" tag there are two "unlock" tags. so for each of the unlock tags the parser calls the unlock_technology function. Our bomber and bomb is now unlocked and able to be crafted. Let's craft the bomber. We click the craft button and the craft_item function traverses the item tree, finds the bomber, deducts all the items needed for crafting, and crafts the item which is placed on our quickbar because our flag mask has the bit for goes-to-quickbar set to 1. Now we have the item in our quickbar but when we try to place it we can't because there is no place result.

So then lets get back to our XML and create one.

Code: Select all

   <place-result>
		<inventory type="storage" size=80 />
		<flags>
			pushable,
			placeable-neutral,
			player-creation
		</flags>
		
		<minable time=1 result="bomber" />
		
		<health max=2000 />
		<corpse type="medium-remnants" />
		
		<selection-box top-left="-0.7, -1.2" bottom-right="0.7, 1.2" />
		
		<acceleration per-energy=0.48 energy-consumption="6W" friction=0.01 weight=250 breaking-speed=0.09>
		
		<inventory type="fuel" size=2 />
		<energy-source type="burn" effectivity=0.5 />
		<emission type="pollution" amount=0.01 />
		
		<smoke deviation="0.1, 0.1" frequency=0.5 position="0, 0" vertical-speed=0.05 >
			<start frame=3 deviation=4 speed=0 speed-deviation=5 />
		</smoke>
		
		<animation filename="__bomber__/graphics/bomber-sheet.png" priority=1>
			<line length = 1 />
			<frame width=211 height=211 shift="0.5, 0" symmetrical="false" directions=9 />
			<rotation speed = 0.01>
		</animation>
		
		<inventory type="storage" size=80 />
		<on-keypress key="q" script="__bomber__/control/drop_bomb.lua" />
	</place-result>

Now we go back into our game. This time, because there is a "place-result" tag to parse, we can place the item in the world. When we do the game opens the XML file and skips to the "place-result" tag and then starts parsing it all the while looking for the relevant parts. At this point we can now start to think of the placed item as an object without a class. This object exists only in the scene graph with it's own unique properties. What are the properties of this object? The first tag the parser come to is "inventory". So a storage type inventory is created with a size of 80. Next the parser sees the "flags" tag and sets the bits in the flag mask for pushable, placeable-neutral, and player-creation to 1. The parser skips the "minable" tag. This is because the item is being placed not mined. Next is the parser hits the "health" tag and sets health to max (2000). Next comes the selection box, which just calls a function to set the data values in a struct. Acceleration is also ignored because we are not accelerating. The next "inventory" tag tells the parser to call a function that creates a fuel inventory of size 2. We aren't moving any where so we don't need to know about the energy source. The animation tag is read just so the starting frame can be set. The parser then moves on to the final inventory and calls a function to create a storage inventory of size 80. The "on-keypress" tag is ignored since no one pressed a key. With the parsing now complete a functioning struct now exists in memory and our bomber appears on screen and it's current state is now idle. Now we click on the plane and, because the parser found 2 inventory tags, we should see a fuel inventory and a storage inventory. We then load the plane with fuel and bombs and hop in. Soon as we press the w key the plane starts to accelerate now the game looks at the XML again and skips to the "accelerate" tag. Now the "acceleration" tag has the "energy-consumption" property, so this tells the parser to look for a source of energy. The "energy-source" tag says that this object burns fuel to accelerate, so the consume_fuel function is called by the parser when that function exits we return to the accelerate function and now the plane begins to accelerate forward. The "animation" and "smoke" tags are parsed and relevant functions are called to animate the place and smoke. The parser also came across an "emission" tag, so every frame the emission function is called and pollution is created. Now we're flying around and we see some biters that we want to bomb. The way the game exists now the closest we can come to the is just waiting for 5 biters to happen to move into range. This time though we have an "on-keypress" tag, so now all we have to do is press a key, in this case q, and the game calls the on_keypress function. Since q does not have a predefined behavior, the XML parser checks the mods for the "on-keypress" tag and executes the associated script. This is the only place where a lua script should exist. "Why?" you ask. Well the answer is simple. The game does not have a drop_bomb function. We need new functionality to drop a bomb. This is also the proper use of a scripting language.

My larger point is this:
I have nothing against lua or C++ classes but every thing has a time and a place. Right now the existing modding system is not very extensible(No XML pun intended). The main reason is the true power of a language like C++ is not being utilized. Everything outside of contorl.lua basically amounts to this:

Code: Select all

bomber = new car();
bomber.name = "bomber";
bomber.stack_size = 1;
bomber.recipe_item1 = 50 iron sticks;
bomber.recipe_item2 = 50 electronic-circuit;

etc

Calling what happens now moding is a bit of a misnomer. To explain the way things are now, lets pretend that Factorio has this class:

Code: Select all

class Apple {
protected
   char *m_color;

public
   Apple(const char *color) {
      strcpy(m_color, color);
   }

   set_color(const char *color, FILE *image) {
      strcpy(m_color, color);
   }

   eat_apple()

  ~Apple()
};

Now lets say we wanted to eat a red apple. Because there is an apple class we can do this:

Code: Select all

(Apple = new Apple("red", apple_pic)).eat_apple();

Now lets eat a green apple.

Code: Select all

(Apple = new Apple("green", apple_pic)).eat_apple();

Now I'm tired of eating apples and I want a banana. Factorio only has apples so I have to "mod" the game to create a banana. I can't create a new class so I write out all the lua code that describes a banana, but there now exists a problem. What would actually happen in C++ code is:

Code: Select all

(Banana = new Apple("yellow", banana_pic)).eat_apple();

It looks like a banana, It's the same color as a banana, but we're still eating apples. No actual banana ever existed in this mod and because of the highly restrictive nature of hard coded classes, and there never will be.

Now lets try things my way. Instead of a class we would have something like this:

Code: Select all

struct Internal_Item_Structure {
   const char *name;
   const char *color;
   FILE *pic;
} item;

void eat_item(item *item);

item *apple = allocate(sizeof(item));
strcpy(apple.name, "Apple");
strcpy(apple.color, "Red);
apple.pic = fopen("graphics/apple_pic.png");

Now we have the same as before but instead of an apple class we just have a struct of item to work with. When we eat the apple, the resulting code is:

Code: Select all

eat_item(apple);

Now lets try to create a banana. We create an XML that describes the banana.

Code: Select all

<item name="Banana" pic="__banana__/graphics/banana_pic.png">
   <color>yellow</color>
</item>

When the XML parser looks at the XML file the resulting behavior in C++ is this:

Code: Select all

item *banana = allocate(sizeof(item));
strcpy(banana.name, "banana");
strcpy(banana.color, "yellow");
banana.pic = fopen("__banana__/graphics/banana_pic.png");

Now that the creation of a new item is complete, lets try to eat the banana;

Code: Select all

eat_item(banana);

This time there are no apples anywhere in sight. We've succeeded in eating a banana.

Now in the real factorio the item struct would be quite large and encompass every variable that any item in the game could possibly need. However, the functions available to the modders would be global and type independent. The sky is the limit not the preexisting classes.

[FreeER]

First, I have relatively little experience with C++ (or other languages) though I can usually understand it when I'm looking at it

Something to keep in mind as you read this is that I have not actually read the source code for the game

If you want to see the implementation of Factorio you are free to ask the devs for access to the source (though they, of course, have the right to deny such requests, they have granted it for a few, 8? I think, people), just send them an email/pm and they'll get back to you.

However, all I see with the xml here is another way to specify the exact same things that lua already does without having to rework how anything is loaded.

The main change that you seem to be suggesting is to use a 'generic struct' rather than 'restrictive classes', but even then you need (all of) the functions to be available (hard coded). What I think you seem to be saying is ditch classes and instead implement a single struct type for every 'thing' (entities and items and tech etc) in the game and give that struct (somehow, can structs have member functions in C++?) all of the functions that could conceivably be used, because it's better to parse a file rather than create/instantiate a class (even though with xml you'd still be using a struct internally instead of a class, so it's not like you are only needing to parse a file and voila an instant object that the C++ code understands. inline edit: that seems a bit sarcastic and rude, sorry it wasn't my intention for it to sound that way!). From my understanding the power of classes is inheritance and polymorphism which structs do not have, of course you pay a price for that ability but it was probably felt that overall such a price was worth being paid. I personally think that classes are a better way to go, because they allow treating entities, items, recipes, and technologies all as separate 'things', which is good because it makes no sense to 'mine' a recipe/item/tech like it does for an entity, just as it makes no (logical) sense to 'craft' an entity or research an entity, you research a tech that unlocks a recipe that is crafted and produces an item that can be placed as an entity which can then be mined (among other things).

The main 'modding' impediment is that the devs didn't/haven't given us access to every single internal function because to do so requires a good deal of work upfront for that access, nor have they found a way to give us the ability to implement our own types directly from lua like you seem to be saying is possible with xml.

Again, due to my limited experience with programming I may be missing what you mean enormously, mainly I can't see how this is realistically possible:

You have pointed out that the main impediment to modding is C++ classes. So lets get rid of those all together.

I actually hope that it is just my lack of experience and the devs can understand exactly what you mean because it would be amazing to have those abilities.

PS. I'm sure reading that (if you managed to read all of it) probably frustrated you severely, sorry

[Zhog]

However, all I see with the xml here is another way to specify the exact same things that lua already does

It is, but because of the way XML can be parsed in C++, it's actually easier on the devs and takes less processor time then executing a lua script. Hence bigger more dynamic mods less load time.

The main change that you seem to be suggesting is to use a 'generic struct' rather than 'restrictive classes'

This is a bit confusing to someone who has never had to use C exclusively. I said get rid of C++ classes, not polymorphic classes. The classes still exist they would just expressed differently, I.E.: 5+2=7, and 2+5=7. It also looks like there is no polymorphism, when in reality the lower you go in C/C++ the more polymorphism you create. This is because, at the lowest level, everything is a void*.

(somehow, can structs have member functions in C++?) all of the functions that could conceivably be used, because it's better to parse a file rather than create/instantiate a class (even though with xml you'd still be using a struct internally instead of a class)

Structs can have member functions, but those functions are public. Memory wise using a struct is much different than using a class. Even though you can't see it, classes have extra functionality behind the curtains that makes a generic struct behave like something more special than what it really is.

I personally think that classes are a better way to go, because they allow treating entities, items, recipes, and technologies all as separate 'things', which is good because it makes no sense to 'mine' a recipe/item/tech like it does for an entity, just as it makes no (logical) sense to 'craft' an entity or research an entity, you research a tech that unlocks a recipe that is crafted and produces an item that can be placed as an entity which can then be mined (among other things).

On some level you are correct: Things need differentiation because without it things won't make sense. however that can also be resolved in a generic way with a programmatic tree. Items go into the item tree, recipes go into the recipe tree, technology goes into the tech tree(and yes these would be a different struct because they are not items), and the items on screen are copied into the scene graph and thus become new independent objects.

The second point about things being generic is also valid, and in the implementation I described you could make a nonsense mod where you eat cars, drive around in trees and mine grass. Such a mod would only be a novelty and the general public would loose interest just as fast as any other fad.

The main 'modding' impediment is that the devs didn't/haven't given us access to every single internal function because to do so requires a good deal of work upfront for that access, nor have they found a way to give us the ability to implement our own types directly from lua like you seem to be saying is possible with xml.

It only seems like it is more work. In the long run it's actually less work. The way the devs have the game implemented in hard coded classes is not ideal for expansion. There is a car class and that car class has it's own functions. Later, lets say, the devs want to make a plane class. A plane is not a car, they don't move the same. A car goes forward, backward, turn left, and turn right. A plane can't inherit these properties because a plane rolls, pitches, and yaws. Only choice left is to create a new class with it's own properties. My way involves only creating a new function (fly(int roll, int pitch, int yaw)), and a new instance of the item struct to describe what the plane looks like, and then expose the fly functionality to the XML parser and thus it becomes available to the moding community. Again this can be used to make a nonsense mod where you fly around the screen in an inserter, and again it would be nothing but a novelty.

PS. I'm sure reading that (if you managed to read all of it) probably frustrated you severely, sorry

I'm not frustrated. I've been a professional programmer for almost 10 years now and have had to explain myself to people with less knowledge about programming than you.

[FreeER]

I'm not frustrated. I've been a professional programmer for almost 10 years now and have had to explain myself to people with less knowledge about programming than you.

Oh good! I think

when in reality the lower you go in C/C++ the more polymorphism you create. This is because, at the lowest level, everything is a void*.

sure, I understand that much, but from what I've heard doesn't that introduce more work by making sure you are manipulating the 'object' (for lack of a better term) properly?

I think I have a (somewhat) better idea of what you are saying now , (and I'll be ignoring the xml vs lua thing here because I understand what you are saying and it's pretty entirely up to the devs from where it is now) but couldn't such things be done similarly by using more generic base/super classes, such as a 'vehicle' instead of just a car and a train etc. such that vehicle defines 'enter/leave vehicle', 'inventory *', 'energy_source *', 'max_altitude', 'can_move_backwards' etc and the car has 'turn left', the train 'set train route', and a plane 'roll/yaw/pitch' (though I think pitch probably makes relatively little difference in a 2D game, roll can easily be part of 'yaw' and 'yaw' is not much more than 'face x degrees' aka 'turn left/right' no? so the real difference between plane and car is possible collisions), then the devs could allow creating a generic 'vehicle' as long as all needed (abstract) functions are provided (though it'd likely be slower unless they managed to implement a method of loading binaries for those functions, which they've mostly chosen not to do at this time for security concerns, I believe)? You could 'drive' an inserter or eat a car now by simply changing the sprite so that's not really the issue I meant to bring up, it was how do you handle calling functions that make no sense (mining/placing a technology or a recipe) without unforeseen results?

Please do correct or otherwise improve my understanding in any and all related to this of course

[Zhog]

sure, I understand that much, but from what I've heard doesn't that introduce more work by making sure you are manipulating the 'object' (for lack of a better term) properly?

Not more work, it's less work, but it's more complex and the devs need to be more vigilant to prevent stack errors.

but couldn't such things be done similarly by using more generic base/super classes,

Yes but consider the impact of a super class. Every time you say new Vehicle() you aren't only copying the functions you need, but also the ones you don't need, thus creating a much larger memory footprint. A struct is much smaller, consisting of only a few bytes a piece. No unneeded functionality is duplicated because a struct would only be variables, the functions should exist in a separate library and be type independent for the most part.

which they've mostly chosen not to do at this time for security concerns, I believe?

The bigger security concern is that they exposed the underlying classes. With simple math I could figure out exactly where they are in memory, and write a virus in C that targets those addresses.

how do you handle calling functions that make no sense (mining/placing a technology or a recipe) without unforeseen results?

First off the technology struct would be a part of the item. I suppose a more realistic representation would be:

Code: Select all

struct INTERNAL_ITEM_STRUCT {
   struct INTERNAL_RECIPE_STRUCT {

  } recipe;

  struct INTERNAL_TECHNOLOGY_STRUCT {

  } technology;
} item;

So you see you couldn't actually place a technology. The research function would still be do_research(item *). However inside that function it would only be looking at the item.technology part. The place function would still be place_item(item *). But inside this function it would only look at things like the picture and the rotation. So like I said before it's more complex type checking but there are no unforeseen results.

[FreeER]

The bigger security concern is that they exposed the underlying classes. With simple math I could figure out exactly where they are in memory, and write a virus in C that targets those addresses.

interesting

you aren't only copying the functions you need, but also the ones you don't need

hm...I was assuming (incorrectly perhaps) that (non overridden) superclass functions were mostly pointers to those functions not a pure copy, and so it wasn't much waste to create those new class objects when you weren't using all of the functions (and since you'd subclass only those related there would be minimal unused functions anyway)...

as for the shown struct(s), wouldn't it be wasting memory by creating a full struct containing (possible) data for all type of objects when you only need the space for a single type? For instance, creating a technology 'item' would only fill out the inner struct for the technology leaving the other structs for recipe/etc 'empty' but wouldn't those empty structs still be allocated memory even though you are not actually using them? Unless of course you meant those inner structs to be struct pointers and not the full struct... which would make it even simpler to check if they are pointing to null I would think...

[Zhog]

creating a technology 'item' would only fill out the inner struct for the technology leaving the other structs for recipe/etc 'empty' but wouldn't those empty structs still be allocated memory even though you are not actually using them? Unless of course you meant those inner structs to be struct pointers and not the full struct...

You are correct sir. I did mean those to be pointers. So if you only wanted a technology:

Code: Select all

item *new_item = allocate(sizeof(item));
new_item.technology = allocate(sizeof(technology));

First notice that no memory was allocated for the recipe. The same is true for any other pointer that may exist in the item struct.
Primitives have a length of one byte. So when you consider the fact that you aren't allocating any item with a length longer than one byte, and even if those bytes are unused you haven't wasted as much memory as a class.

[FreeER]

Primitives have a length of one byte.

...the only thing that I know of that is one byte (at least typically) is char (and perhaps some typedefs), are not the rest (int for example, which is used to store pointers) typically 4 bytes on a x86 system and 8 bytes on a x64 system (of course then you have real numbers but I'm not sure if those are counted as primitives or not)? Not to mention possible padding for alignment (unless specified as packed, but that loses access performance for minimal memory savings), so wouldn't the real size of an item struct be 4 * 4 bytes + possible padding (on x86) or 4 * 8 bytes + possible padding (on x64)? (four/eight bytes for each stored pointer (whether it points to valid data or null): item (current version of it, hm..a better name for the base struct is probably 'prototype'), recipe, entity, technology), plus of course the actual size of the implemented struct (though that will vary, with storage of file-paths to images and length of names etc, not to mention the actual loaded resources like the binary images).

Also, googled and found this: which basically says that by the language standards structs are just classes with members specified as public by default and classes are structs with default private members...so is there really a performance difference there except for the inheriting (which must use at least a pointer to the superclass if not more), which according to the referenced forum can also be done with structs (though I've never seen it, I haven't seen that much code either lol)? Of course it could be wrong (I'm at least pretty sure I didn't miss read it )

also, 'nit-picky' but it's probably somewhat important:

new_item.technology = allocate(sizeof(technology));

should probably be

Code: Select all

new_item->technology = allocate(sizeof(technology));

[cube]

That generic XML thing is pretty close to a "LuaEntity" we have in mind for some time.

The main problem in creating entities like this is specifying their behavior, not recipes or technologies (these are already handled almost exactly in the way you described).
For example the bomb has to know when to explode, bomber has to know how many bombs it holds and that the number should increase when docking (or something like that). This is very generic and cannot be done without a full programming language (Mr. Turing said something like that )

Currently most of the behavior is coded in C++ for performance reasons. You can use LuaJIT, or whatever other tricks, but it never will be quite as fast as compiled and optimized native code. This means that even in the future we want most of the factory run of hard coded entities, but for small number of special entities it is acceptable to pay the price of having behavior in interpreted language and that's where the LuaEntity comes to play.

It would be very simillar to what you described, only using lua for everything instead of XML. This kind of entity would have a set of hooks where you can put custom lua code and the engine wouldn't do almost anything at all with the entity by itself....

And finally: These are some really nice plans, but we have more important stuff to add to the game in the next few months

[Zhog]

Primitives have a length of one byte.

...the only thing that I know of that is one byte (at least typically) is char (and perhaps some typedefs), are not the rest (int for example, which is used to store pointers) typically 4 bytes on a x86 system and 8 bytes on a x64 system (of course then you have real numbers but I'm not sure if those are counted as primitives or not)? Not to mention possible padding for alignment (unless specified as packed, but that loses access performance for minimal memory savings), so wouldn't the real size of an item struct be 4 * 4 bytes + possible padding (on x86) or 4 * 8 bytes + possible padding (on x64)? (four/eight bytes for each stored pointer (whether it points to valid data or null): item (current version of it, hm..a better name for the base struct is probably 'prototype'), recipe, entity, technology), plus of course the actual size of the implemented struct (though that will vary, with storage of file-paths to images and length of names etc, not to mention the actual loaded resources like the binary images).

This is what happens when you drink too much and try to program. Never do that.

These are some really nice plans, but we have more important stuff to add to the game in the next few months

I understand your position completely. Much of what I proposed would require a lot of work.

[FreeER]

This is what happens when you drink too much and try to program. Never do that.

I've never done it in C++ but I have in Lua...came back the next morning and had to redo practically all of the code I've never done so since, once I start drinking (which isn't often to be honest) I stop programming cause it's just not worth the hassle of trying to figure out what drunk me thought I was doing

[Schmendrick]

This is what happens when you drink too much and try to program. Never do that.

I've never done it in C++ but I have in Lua...came back the next morning and had to redo practically all of the code I've never done so since, once I start drinking (which isn't often to be honest) I stop programming cause it's just not worth the hassle of trying to figure out what drunk me thought I was doing

I think the key part is "too much." It's a joke, but there is some truth to xkcd's Ballmer Peak. One time, I was pulled from an interesting/fun embedded driver in vanilla ANSI C to work on a cludgy, twice-previously abandoned XML based server abomination in C# (which I'd never used before) and told it had to be in production by the end of the next week (fortunately, the deadline ended up being flexible). Perhaps it was the stress, the rapid mental shift I needed to perform in project type, scope, and environment, and/or some real life Ballmer Peak in action, but whatever it was, alcohol definitely helped. All programmers should be allowed to keep a bottle of whiskey at their desk.

As a side note to the devs, just more extensive lua level access to game mechanics would be nice; any baked-in formulas (especially stuff that should be performed every tick) used by existing entities. For example, making components of entities more generic, like using sunlight levels or fluid-temperature-variance <-> energy conversion outside the specific functionality of the solar panels, steam engines, and boiler types would allow things like electric boilers and endo-/exo-thermic reactions in chemical plants (or whatever) be a thing. Having a generic entity with a several possible canned input energy types (electricity, fuel, temperature [variance], solar, wind) and a few output types (electricity, crafting, fluid temperature), or add a quick way to add heat leakage to pipes and tanks (which would also suggest a specific heat value for fluids, but I digress). Basically, instead of opening up everything to lua access, simply breaking down the basic conversions in the game (each probably taking an effectivity modifier such as in the boiler) and making a generic object that is C++ executed (and thus efficient) but lua "wired" would be great, and probably cut down on a lot of mods' ontick subscriptions, or at least the activity therein.

[robhol]

XML is actually not a very good language. It has its uses, but JSON or Lua-style tables are arguably much more appealing, definitely less space-wastey, less verbose, not to mention they already fit in with the game engine.

The main problem with the definitions now is that it's somewhat/very unclear which attributes do what and which of them will play nice together.
I'd really like the idea of just defining "interfaces" like you touched upon in your post. By adding the "fluid" interface, you could make tanks, pumps, pipes and so on. By adding the "electricity" interface, you'd be able to define power consumption and so on. More importantly, they'd be able to work together, instead of the current solution where (as far as I can tell with my limited modding experience) the "type" of an entity decides what it can do instead of you just telling the engine what it can do.

[nullvoid]

Food for Thought: This (Interfaces) is similar to making Factorio work on an entity component system, i.e. composition rather than inheritance.

Though it's probably a bit late in the game to switch paradigm like that.

[cube]

Food for Thought: This (Interfaces) is similar to making Factorio work on an entity component system, i.e. composition rather than inheritance.

Though it's probably a bit late in the game to switch paradigm like that.

+1

Also I'd think that entity systems will tend to have worse prerformance.

[Rahjital]

It depends. Entity systems can be optimized to be nearly as fast as inheritance systems at the cost of making them somewhat unintuitive to use. Factorio is way too far in development to actually use them, since pretty much everything would have to be rewritten from scratch, including the entire entity and systems. The only ones who would truly benefit from this would be modders, since there would be nothing preventing them from creating even unusual things like enemy units mixed with accumulators that would find an undefended power pole and drain energy from the player's power network.

[JamesOFarrell]

I've been thinking about how to solve this problem without moving every thing away from an inheritance based system and there is a simple solution that could get us over this hump. At the moment we can sort of hack these solutions together using multiple transparent entities to get access to their functionality in where we should not have it. This works really well but a lot of time is dedicated to doing simple things like tracking the onbuild event for the entity being created. Why not embrace this and give us an entity group that is just a reference to a parent and some children. The children are slaved to the parent for location, direction, health and but only the parent is rendered. The children exist in the game and can provide their usual functionality but can't be interacted with or seen by the player. This allows modders to mix and match functionality without moving the whole game over to a new design philosophy and refactoring the codebase. I know is seems like a hack and the clean solution is to implement interfaces but that just seems untenable with the multiplayer work also going on. Thoughts?