Friday Facts #209 - Optimisation is a way of life

Regular reports on Factorio development.

Re: Friday Facts #209 - Optimisation is a way of life

Postby hoho » Mon Sep 25, 2017 7:59 pm

Heh, IPC is pretty much the definition of how good a CPU is at running stuff, at least when it's measured in practice and not just something on paper*. Yeah, Intel has been better than AMD since Core/Core2 came out and K10 flopped. HUGE part of it is significantly better cache/memory system that is able to get data to the CPU more effectively.

*) I remember the time when PS3/XB360 came out and sony-MS were competing on who has the highest "ipc" in their system. They came out with some hilarious numbers such as 2 TFLOPS while less than 10% of that was actually programmable and even that was just on paper. Real-world numbers were somewhere near 100GFLOPS at best when combining the CPU and GPU. Sony counted HW-based stuff like trilinear texture filtering while MS was using similarly HW-based functionality in their embedded-DRAM chip that had HW-based anti-aliasing.
hoho
Filter Inserter
Filter Inserter
 
Posts: 704
Joined: Sat Jan 18, 2014 11:23 am

Re: Friday Facts #209 - Optimisation is a way of life

Postby _alphaBeta_ » Mon Sep 25, 2017 8:40 pm

Kno wrote:I think you are going attacking the bot dilemma the wrong way. First you have to ask if logistics robots being an entity is a desirable game-play element, then see how to optimise the implementation, and not the other way around. Many gameplay elements could be eliminated for the sake of a faster game, but that is not optimising anything, that's eliminating parts.

Going to the extreme to fix the UPS problem of robots, why not add a late game tech that adds teleport chests, that way there is no graphics, not robot position or anything, just pure optimised item transfers.

Neemys wrote:If you just change logistic and not construction bot, it will be weird to have two bots acting so differently (not interactable and invulnerable for one).

And having things that can move resource even over enemy base is not in line with other transportation mode. As bot always fly there will be no way for biter to stop bot from transporting as even destroying roboport won't stop them to transport. Belts and train can be destroyed. In some use case it can be use like an exploit even more in some modded game with big roboport that can cover enemy base.


I agree with this.


I'd have to fully agree that gameplay should not be sacrificed for optimization. The only way I would even begin to support this change is if bots stay within their construction zones, and not cut corners over enemy territory. This would reduce, but not eliminate, the impacts in my mind.

Bottom line, I can't get excited about this change unless additional options and/or optimizations discussed in this thread concerning how bots work are also implemented.
User avatar
_alphaBeta_
Inserter
Inserter
 
Posts: 29
Joined: Fri Jul 29, 2016 3:27 am
Location: NJ, USA

Re: Friday Facts #209 - Optimisation is a way of life

Postby TheRaph » Mon Sep 25, 2017 9:32 pm

Kno wrote:I think you are going attacking the bot dilemma the wrong way. First you have to ask if logistics robots being an entity is a desirable game-play element, then see how to optimise the implementation, and not the other way around. Many gameplay elements could be eliminated for the sake of a faster game, but that is not optimising anything, that's eliminating parts.

Going to the extreme to fix the UPS problem of robots, why not add a late game tech that adds teleport chests, that way there is no graphics, not robot position or anything, just pure optimised item transfers.

Neemys wrote:If you just change logistic and not construction bot, it will be weird to have two bots acting so differently (not interactable and invulnerable for one).

And having things that can move resource even over enemy base is not in line with other transportation mode. As bot always fly there will be no way for biter to stop bot from transporting as even destroying roboport won't stop them to transport. Belts and train can be destroyed. In some use case it can be use like an exploit even more in some modded game with big roboport that can cover enemy base.


I agree with this.

I also agree with this.

Additionally I think bots don't need to be minable (a request button in roboport should be a better alternative), because this behaviour gets messy in big bases. But if bots don't take damage at all, this won't fit the gameplay.

I've read here some suggestions about switching bots between "smoke-mode" and "normal mode" to have them optimized in most of the time when they do their job and be destroyable if necessary. For me, that sounds great. (If combined with "not minable" as mentioned above the smoky bots only need to check if enemies are nearby, the presence of the player - or multiple players on servers - don't need to be checked. )

I'm not a programmer, but would it be possible to use a smoke-like update chain as described in FFF only for positioning on map and use a pointer to the (elsewhere stored) entity if it comes to interactions (like load, unload, battle etc.)?
TheRaph
Inserter
Inserter
 
Posts: 34
Joined: Sun Sep 24, 2017 6:31 pm
Location: Pinneberg, Northwestern Germany

Re: Friday Facts #209 - Optimisation is a way of life

Postby bobucles » Tue Sep 26, 2017 1:06 pm

If bots aren't minable, then a situation needs to be resolved first. If a logistic network is cut in half, bots can end up stranded floating around like dummies. There's no clean way to fix the problem if the bots can't be picked up (maybe the network is meant to be split and got merged by accident).
bobucles
Filter Inserter
Filter Inserter
 
Posts: 723
Joined: Wed Jun 10, 2015 10:37 pm

Re: Friday Facts #209 - Optimisation is a way of life

Postby Sonik-HSC » Wed Sep 27, 2017 12:17 pm

The stone path is beautifull..... But there is one litle problem.... when he put stone or concrete path they don´t cover the vegetation.... and the minerals on flor....
This can not be done??? its a litlle feature that is missing " cover completely the floor " But everything is nice ! :mrgreen:
Sonik-HSC
Inserter
Inserter
 
Posts: 36
Joined: Wed Feb 08, 2017 2:06 am

Re: Friday Facts #209 - Optimisation is a way of life

Postby TurkleTrenox » Wed Sep 27, 2017 6:57 pm

I´m thinking of make me an account on those sites where you get money from people around the web to buy a motherboard whit more slots and a graphic card to play the game properly and upload the videos to youtube; if a do that, will update the comment whit "zeldas"
TurkleTrenox
Burner Inserter
Burner Inserter
 
Posts: 11
Joined: Sat Jul 08, 2017 12:16 am

Re: Friday Facts #209 - Optimisation is a way of life

Postby WarStalkeR » Thu Sep 28, 2017 10:37 am

kovarex wrote:https://www.factorio.com/blog/post/fff-209

Hello Kovarex, as True God of Optimization, can you please post some short tips for optimization for Stellaris developers? Either they have hard time or just can't into optimization as they should (as almost everybody else). Just post these tips here and I will make sure that they will see them one way or another. Thanks!
WarStalkeR
Burner Inserter
Burner Inserter
 
Posts: 10
Joined: Sat Nov 01, 2014 9:10 am

Re: Friday Facts #209 - Optimisation is a way of life

Postby hoho » Thu Sep 28, 2017 11:56 am

WarStalkeR wrote:
kovarex wrote:https://www.factorio.com/blog/post/fff-209

Hello Kovarex, as True God of Optimization, can you please post some short tips for optimization for Stellaris developers? Either they have hard time or just can't into optimization as they should (as almost everybody else). Just post these tips here and I will make sure that they will see them one way or another. Thanks!

I'm no kovarex but the high-level ideas for low-level optimizations applies equally to pretty much every code regardless of language or CPU used.

*) Before starting with *any* optimizations, run your code in a profiler to see what really takes the most time. There are tools that use special CPU instructions to profile your code without causing massive overhead. VTune is probably the best but costs a ton. MSVC also has something similar and is free
*) Before looking for chances of low-level optimizations, see if you could use more efficient algorithms
*) Try to allocate-free memory rarely as possible. Both are VERY expensive operations.
*) keep your data structures together in memory (arrays instead of linked lists wherever it makes sense)
*) if you have to iterate over the same array multiple times, try to replace it with a single iteration where you do all the operations at same time
*) Conditional branching (if then else) is relatively expensive, try to minimize them. If you can replace an if with a simple-ish calculation, it usually makes sense to do it.
*) Lookup tables are often but not always cheaper than re-calculating stuff as needed. Just make sure they're not big enough to trash the cache
*) Using binary search in arrays of ~100 elements is slower than linear iteration over it assuming check is relatively simple (checking for boolean or a couple of numeric values). In addition, you won't have to keep it sorted to even allow for binary search.
*) Use the smallest data types you can get away with (32bit int instead of 64, float instead of double)
*) If you use SIMD anywhere, make absolutely sure that your arrays are cache line aligned and preferrably get your data structures to be small enough to fit in a cache line.
*) Object oriented programming with class hierarchies and virtual functions usually makes code easier to write but calling virtual functions can be extremely expensive. Similarly, c++ templates can generate a huge amount of code that ends up trashing your memory.
*) Move as much stuff out of loops as you can. E.g if you loop over an array and check its size as the end condition, that check can be quite expensive if it's some kind of a relatively complex thingy (e.g child object inside a virtual class). I have had a case where moving such a check out from loop header made the *overall* application run several percentages faster and the program was anything but trivial.
*) Most modern languages have some basic forms of code parallelization built-in (usually some kind of parallel for loop). If you have calculations you perform on a ton of objects that don't have side effects, using those can give some "free" performance.
*) When working with arrays with more than one dimension, make sure you iterate over the elements in the order they are laid out in memory.
*) When filling arrays with data, try to figure out their final size before allocating them. Doing x.append(y) or x.add(y) can become awfully slow when the library has to reallocate the buffer every time it runs out of space.


There are a ton more tricks but those are some of the things that come to mind right now. Of these, profiling and figuring out better algorithms are by far the most important ones.
hoho
Filter Inserter
Filter Inserter
 
Posts: 704
Joined: Sat Jan 18, 2014 11:23 am

Re: Friday Facts #209 - Optimisation is a way of life

Postby Meddleman » Fri Sep 29, 2017 11:53 am

kovarex wrote:The electric network optimisation for 0.16 was simple in principle. We currently have one continuous buffer of electric connector data stored in the electric network. It needs to be iterated twice (first for calculation, second for power distribution).


This is a bit of a topic segue, but does the electric network measure where the electric energy is coming from? Or does it simply just check each closed web every tick and cache all the electricity producers/consumer in one go?
The point I am trying to make is that while the map shows us a nice highlighted spiderweb of connected power lines, it doesn't highlight the webs that are without power. Yes I know this has been suggested before so I'll not nitpick about that detail.
However too often there are "critical connection poles"; those poles that essentially connects different webs together to create a larger electric network. A favourite hunting game for griefers on multiplayer, wishing to employ the maximum ratio of chaos vs. effort/detection risk to create said chaos.

My final question is thus:
If the game already caches seperate electrical power webs and their contained energy producers/consumers, is it able to pinpoint specific poles that connect what would have once been two or more smaller webs into a larger one, and if so, can this be visualised? Perhaps a flashing of the cyan-lines/blue power pole dots or gradiental hue change to show bottlenecks amongst said webs.

I understand that on a single line of a near infinite line of poles, with a consumer on one end, and a producer on the other, what factor decides the "critical pole"? Simple answer, there isn't one because they all are. With this in mind I understand the complications in pinpointing any specific "critical connection" pole.

This is not a change I would expect to see within the next few months or years, and god knows there are a boatload of critically more important issues to address. I figured that if power network optimization is being tackled for 0.16, I'd throw in my two cents to mull over briefly.
Attachments
example.png
example.png (56.93 KiB) Viewed 577 times
Meddleman
Burner Inserter
Burner Inserter
 
Posts: 7
Joined: Mon Jun 26, 2017 7:39 pm

Re: Friday Facts #209 - Optimisation is a way of life

Postby bobucles » Mon Oct 02, 2017 5:58 pm

I like the idea of color coding electrical networks. I wonder if it would be more useful to give each network its own color, or if it would be better to color each network according to its electric satisfaction? It is very common that a disconnected power network gets into the red, so it'd still be color coded where it matters.
bobucles
Filter Inserter
Filter Inserter
 
Posts: 723
Joined: Wed Jun 10, 2015 10:37 pm

Re: Friday Facts #209 - Optimisation is a way of life

Postby hitzu » Thu Oct 12, 2017 1:36 pm

Image
How does the stone interact with the concrete?
Now the stone looks sunken. New graphics with border stones give the impression it should be raised above the concrete.
I have no better example
Image
hitzu
Filter Inserter
Filter Inserter
 
Posts: 490
Joined: Tue Sep 09, 2014 5:55 pm

Re: Friday Facts #209 - Optimisation is a way of life

Postby mrvn » Thu Oct 12, 2017 3:29 pm

cpy wrote:Well how about allowing mods to use separate CPU thread to run? I'm sure that factorisimo running in 8 more threads itself with each of mini factory which is world itself would allow us to create insanely big worlds. But that would require some connection that connects those separated worlds.


Most mods interact with the world and running them in parallel would require synchronization without end.

But there are some mods like Factorissimo that create separate maps with very limited interaction with the main map through LUA functions. Instead of running mods in parallel could those separate maps be run in parallel and then when all maps are finished run the mods for the part where it causes interactions between the maps?
mrvn
Smart Inserter
Smart Inserter
 
Posts: 1334
Joined: Mon Sep 05, 2016 9:10 am

Re: Friday Facts #209 - Optimisation is a way of life

Postby bobucles » Fri Oct 13, 2017 2:21 am

But there are some mods like Factorissimo that create separate maps with very limited interaction with the main map through LUA functions. Instead of running mods in parallel could those separate maps be run in parallel and then when all maps are finished run the mods for the part where it causes interactions between the maps?

Allowing all the things that can go wrong with multi threading, for the one narrow thing that might go right? That level of complexity can not be predicted by a simple mod engine, at least not without also handing out a nobel prize. The mod would have to be very carefully set up at a deep level in order to have any chance of working without heaps of issues.

Don't get me wrong, it'd be pretty neat if factorio could do the whole "parallel world" thing in a way that lets the game outsource processing power. If it can let another computer carry the load, then lots of computers can do it and that means bigger factories than ever before. But something like that carries its own box of issues that need to be overcome. It's definitely too big to treat like something that can go into an ordinary patch.
bobucles
Filter Inserter
Filter Inserter
 
Posts: 723
Joined: Wed Jun 10, 2015 10:37 pm

Re: Friday Facts #209 - Optimisation is a way of life

Postby mrvn » Mon Oct 16, 2017 11:27 am

bobucles wrote:
But there are some mods like Factorissimo that create separate maps with very limited interaction with the main map through LUA functions. Instead of running mods in parallel could those separate maps be run in parallel and then when all maps are finished run the mods for the part where it causes interactions between the maps?

Allowing all the things that can go wrong with multi threading, for the one narrow thing that might go right? That level of complexity can not be predicted by a simple mod engine, at least not without also handing out a nobel prize. The mod would have to be very carefully set up at a deep level in order to have any chance of working without heaps of issues.

Don't get me wrong, it'd be pretty neat if factorio could do the whole "parallel world" thing in a way that lets the game outsource processing power. If it can let another computer carry the load, then lots of computers can do it and that means bigger factories than ever before. But something like that carries its own box of issues that need to be overcome. It's definitely too big to treat like something that can go into an ordinary patch.


That's why I said it should run the maps, the core stuff like assembler, pipes, belts and inserters. All the stuff done in C++ and known factors. Then at the end when all the known stuff has completed in parallel you run the LUA callbacks from the mods single threaded.
mrvn
Smart Inserter
Smart Inserter
 
Posts: 1334
Joined: Mon Sep 05, 2016 9:10 am

Re: Friday Facts #209 - Optimisation is a way of life

Postby Koltrast » Mon Oct 16, 2017 12:44 pm

I'm sorry but, the pavement's border looks awful.

The player want to leave the alien planet, not furnish and beautify it... imho it should look more rattled.
Koltrast
Inserter
Inserter
 
Posts: 25
Joined: Sun Oct 09, 2016 9:40 pm

Re: Friday Facts #209 - Optimisation is a way of life

Postby mrvn » Mon Oct 16, 2017 2:30 pm

Koltrast wrote:I'm sorry but, the pavement's border looks awful.

The player want to leave the alien planet, not furnish and beautify it... imho it should look more rattled.


To me the border looks like it is floating just above the surface. It should be sunken into the sand or earth a bit. Make the ground uneven so it is sunken in differently at different points.
mrvn
Smart Inserter
Smart Inserter
 
Posts: 1334
Joined: Mon Sep 05, 2016 9:10 am

Previous

Return to News

Who is online

Users browsing this forum: Omarflyjoemacky, pieinvestigator and 8 guests