Friday Facts #270 - HR Substation & Save/Load overview

Regular reports on Factorio development.
User avatar
Dixi
Fast Inserter
Fast Inserter
Posts: 141
Joined: Sat Nov 04, 2017 1:47 pm
Contact:

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by Dixi »

I always thought that "Entities", if biters fit in there, takes much more space in a save file.
Because, when everything is calm, my save file is around 40Mb. If I start to expand and clear a new map piece, even not so big one, or do some manual artillery shooting (both actions leads to intensive biters attack waves) save easily can grow to 60+Mb. If I delete all alive biters by console command save instantly shrinks to 25Mb. ( 15+20 )/ 60 is over 50% of save file!
I've tried to complain on forums before, that biters, while been just a side factor in a game about factories take too much save file space. I complain, because it's inconvenient to have very big save, because it freezes the game every several minutes for autosave. Maybe it's possible to review possibility to copy data, then save without game pause? I remember, it was rejected for some reason, but I still think it could be an important improvement to gameplay smoothness.

ske
Filter Inserter
Filter Inserter
Posts: 411
Joined: Sat Oct 17, 2015 8:00 am
Contact:

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by ske »

Dixi wrote: ↑
Sun Nov 25, 2018 10:57 am
it was rejected for some reason
I'd like to know that reason.

Most likely running two server/simulation processes on most processors reduces maximum UPS considerably since factorio performance is said to be memory bound. Also doubling the background logic could be somewhat complicate things. But I think now it would be worth it to support massive multiplayer games without those pesky pauses every 10 seconds when somebody tries to (re)join.

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 5206
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by eradicator »

ske wrote: ↑
Sun Nov 25, 2018 2:25 pm
Dixi wrote: ↑
Sun Nov 25, 2018 10:57 am
it was rejected for some reason
I'd like to know that reason.

Most likely running two server/simulation processes on most processors reduces maximum UPS considerably since factorio performance is said to be memory bound. Also doubling the background logic could be somewhat complicate things. But I think now it would be worth it to support massive multiplayer games without those pesky pauses every 10 seconds when somebody tries to (re)join.
In server options/config you can enable non-blocking saving, though it seems it doen't work on windows?
nonblock.png
nonblock.png (38.54 KiB) Viewed 6434 times
Author of: Belt Planner, Hand Crank Generator, Screenshot Maker, /sudo and more.
Mod support languages: ζ—₯本θͺž, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.

factoriouzr
Filter Inserter
Filter Inserter
Posts: 660
Joined: Sat Jun 06, 2015 2:23 am
Contact:

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by factoriouzr »

So can we have a release of 0.17 before the holidays this year? It's been a ridiculously long time since the last Factorio release; almost a year in fact. 0.16.0 was released Dec 13 2017. A lot of us will have some time to play over the holidays and a release of 0.17 before the holidays would be great.

Wube Software said that you would release more frequently, however this definitely doesn't seem to be the case.

I appreciate all the work you are doing, but what goes into each release is arbitrary. You can release the features you have ready now and have time to fix critical bugs before holiday vacation. Then you can fix the rest of the issues after holiday vacations (I don't expect people to not take time off during the holidays because of a release). Then you can release the rest of the planned feature for 0.17 in January as you mentioned you wanted to.

pleegwat
Filter Inserter
Filter Inserter
Posts: 252
Joined: Fri May 19, 2017 7:31 pm
Contact:

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by pleegwat »

ske wrote: ↑
Sun Nov 25, 2018 2:25 pm
Dixi wrote: ↑
Sun Nov 25, 2018 10:57 am
it was rejected for some reason
I'd like to know that reason.

Most likely running two server/simulation processes on most processors reduces maximum UPS considerably since factorio performance is said to be memory bound. Also doubling the background logic could be somewhat complicate things. But I think now it would be worth it to support massive multiplayer games without those pesky pauses every 10 seconds when somebody tries to (re)join.
Not memory bandwidth, memory latency. So that's not the immediate problem.

However it's built single-threaded; an action taken earlier in the tick (EG an inserter picking up an item from the ground) can cause an action later in the same tick (EG another inserter attempting to pick up the same item) to fail.

And I think the pauses are caused by the server saving current game state, so I don't see how multithreading would mitigate that.

User avatar
5thHorseman
Smart Inserter
Smart Inserter
Posts: 1193
Joined: Fri Jun 10, 2016 11:21 pm
Contact:

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by 5thHorseman »

factoriouzr wrote: ↑
Sun Nov 25, 2018 2:55 pm
So can we have a release of 0.17 before the holidays this year?
They answered that already.

No.

ske
Filter Inserter
Filter Inserter
Posts: 411
Joined: Sat Oct 17, 2015 8:00 am
Contact:

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by ske »

eradicator wrote: ↑
Sun Nov 25, 2018 2:44 pm
In server options/config you can enable non-blocking saving, though it seems it doen't work on windows?
nonblock.png
Interesting, I wonder what technology it's using.

posila
Factorio Staff
Factorio Staff
Posts: 5200
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by posila »

ske wrote: ↑
Sun Nov 25, 2018 6:05 pm
eradicator wrote: ↑
Sun Nov 25, 2018 2:44 pm
In server options/config you can enable non-blocking saving, though it seems it doen't work on windows?
nonblock.png
Interesting, I wonder what technology it's using.
It's forking the factorio process, that's why it doesn't work on Windows (no official fork on Windows, and we didn't managed to make it work with the un-official undocumented one)

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 5206
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by eradicator »

posila wrote: ↑
Sun Nov 25, 2018 6:14 pm
ske wrote: ↑
Sun Nov 25, 2018 6:05 pm
eradicator wrote: ↑
Sun Nov 25, 2018 2:44 pm
In server options/config you can enable non-blocking saving, though it seems it doen't work on windows?
nonblock.png
Interesting, I wonder what technology it's using.
It's forking the factorio process, that's why it doesn't work on Windows (no official fork on Windows, and we didn't managed to make it work with the un-official undocumented one)
Wouldn't hurt if there was a short tooltip and description in the config that says "Only on Mac/Unix.". Nothing more confusing than a checkbox that doesn't do anything :p. Thanks for the explanation ofc though.
Author of: Belt Planner, Hand Crank Generator, Screenshot Maker, /sudo and more.
Mod support languages: ζ—₯本θͺž, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.

posila
Factorio Staff
Factorio Staff
Posts: 5200
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by posila »

Honestly, I thought the checkbox doesn't show up on Windows.

ske
Filter Inserter
Filter Inserter
Posts: 411
Joined: Sat Oct 17, 2015 8:00 am
Contact:

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by ske »

posila wrote: ↑
Sun Nov 25, 2018 6:14 pm
ske wrote: ↑
Sun Nov 25, 2018 6:05 pm
eradicator wrote: ↑
Sun Nov 25, 2018 2:44 pm
In server options/config you can enable non-blocking saving, though it seems it doen't work on windows?
nonblock.png
Interesting, I wonder what technology it's using.
It's forking the factorio process, that's why it doesn't work on Windows (no official fork on Windows, and we didn't managed to make it work with the un-official undocumented one)
I see. Is copy-on-write quick enough to avoid significant lag when allocating and copying all that memory? (I assume that all pages are touched by some write access.)

If it is sufficient, could you rewrite the game logic in such a way, that it creates a separate data structure for each frame in a new memory area?

posila
Factorio Staff
Factorio Staff
Posts: 5200
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by posila »

ske wrote: ↑
Sun Nov 25, 2018 7:29 pm
I see. Is copy-on-write quick enough to avoid significant lag when allocating and copying all that memory? (I assume that all pages are touched by some write access.)
I heard, the game stutters while saving kicks in.
ske wrote: ↑
Sun Nov 25, 2018 7:29 pm
If it is sufficient, could you rewrite the game logic in such a way, that it creates a separate data structure for each frame in a new memory area?
So 0.17 sometime in 2020? :)

ske
Filter Inserter
Filter Inserter
Posts: 411
Joined: Sat Oct 17, 2015 8:00 am
Contact:

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by ske »

posila wrote: ↑
Sun Nov 25, 2018 7:33 pm
ske wrote: ↑
Sun Nov 25, 2018 7:29 pm
If it is sufficient, could you rewrite the game logic in such a way, that it creates a separate data structure for each frame in a new memory area?
So 0.17 sometime in 2020? :)
:o I hope I didn't give you any new ideas. I mean I think it could probably be done. What the operating system is doing during COW is not much more than a simple memory copy. If you do that manually you should be more efficient. You'd just have to use an allocator for everything relevant for the game state and then just copy all memory areas allocated by that allocator. That's a few memory areas. Sure you'd need to fix the pointers but that could be done by some lookup-tables or a simple offset if you only have one container. Sure, you'd need to apply that offset to all internal pointers but that could be covered by using templates that optimize away such an offset for the normal operation. After copying you'd only need to serialize the whole package and discard it.

FasterJump
Fast Inserter
Fast Inserter
Posts: 188
Joined: Sat Jul 09, 2016 11:43 am
Contact:

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by FasterJump »

I know that chunks that are far away (and no polluted) are inactive (frozen) to save fps.

Would it be possible to delete specific chunks automatically, to save map size? Like a chunk where no nest was destroyed and no ore mined. The minimap stays updated, but the chunk is deleted as soon as the fog war comes back.

User avatar
bobingabout
Smart Inserter
Smart Inserter
Posts: 7351
Joined: Fri May 09, 2014 1:01 pm
Contact:

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by bobingabout »

Rseding91 wrote: ↑
Fri Nov 23, 2018 7:19 pm
bobingabout wrote: ↑
Fri Nov 23, 2018 6:58 pm
I don't really like the new substation graphic, but I'm sure it will grow on me.

The save file is more than a quarter decoratives? Most games don't even save those, just re-generate them on load, or when you re-enter an unloaded area. I don't see why you can't do the same. Is there any reason these need to remain perfectly in sync?
Determinism.
do "Decoratives" really matter though? I remember playing Terraria, and moving around, and when I come back the decoratives have changed their appearance, because they aren't actually saved. So two players in a multiplayer can be looking at the same spot, and see different things because the decoratives are different, it really doesn't matter.

So, my question is basically... is it REALLY important enough to save? would it break the game if they weren't perfectly in sync? 30% of a save seems like a lot of space to dedicate to something that most people just see as annoying (because it hides more important things from view, or look ugly EG grass coming through your concrete floor)

Don't get me wrong, I don't think these "annoying" doodads should be removed, they add character, I just don't see why they need to be synced in the name of determinism.
IronCartographer wrote: ↑
Sat Nov 24, 2018 12:01 pm
Regarding the HR graphic update and complaints that it looks too small / noisy / similar to a power pole...
Image

I agree that it no longer looks like something I would call a substation, though the new graphic is nice in its own right. What if the mismatch can be resolved while using the new art?

How about a new name: Electric Transceiver! It fits because of the huge wireless coverage area, both sending and receiving power.

Parallel discussion on Reddit: https://www.reddit.com/r/factorio/comme ... ?context=2
Also it would seem the diagonal pole rotations are backwards.
Creator of Bob's mods. Expanding your gameplay since version 0.9.8.
I also have a Patreon.

mrvn
Smart Inserter
Smart Inserter
Posts: 5646
Joined: Mon Sep 05, 2016 9:10 am
Contact:

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by mrvn »

eradicator wrote: ↑
Fri Nov 23, 2018 8:23 pm
Oktokolo wrote: ↑
Fri Nov 23, 2018 7:44 pm
eradicator wrote: ↑
Fri Nov 23, 2018 7:21 pm
For example concrete has a random chance to destroy decoratives. Could probably be made deterministic.
Random numbers are already deterministic in Factorio. They are based on synced state.
I doubt you understand what you just said there. The results of random numbers generated during runtime are part of the savegame in the form of decorative positions. Not saving decoratives would mean recalculating all random numbers from the start of the save (i.e. running a full replay before every load) or completely changing how concrete determines which decoratives are removed.
______
Koub wrote: ↑
Fri Nov 23, 2018 8:05 pm
Most of the time, I wish they weren't there, but I can't take them off without a mod, which would in turn prevent me from earning achievements on Steam.
Steam "achievements" the greatest non-reason ever invented to play a game you like less instead of a game you like more :roll:.
______
I was never very fond of the current substation, always looked too...flat? modern? Didn't have that "steampunk" feel written all over it. Not that the new one couldn't need some more of that too.
No, you would use a separate random number generator for decorations so they wouldn't affect the numbers of anything else.

But why do decoration need to be random at all? Why not make them noise instead? The "drawback" would be that when you have e.g. a leached animal skull in the desert, change that to grass and back to desert then the leached animal skill would be back. Because the noise function says there should be one. Weather that is a plus or minus is up to you.

PS: When you landfill an area you kind of see the stripes of landfill you placed because going up and down look different. Play Seablock for a while and you will notice it.

weaknespase
Long Handed Inserter
Long Handed Inserter
Posts: 59
Joined: Sat Mar 24, 2018 8:19 am
Contact:

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by weaknespase »

Nonblocking saving? Yeah, forking won't work most probably, due to the mechanics of how forking works. The process that modifies dataset will get all the usual goodness - page faults, copying of memory, thousands of page allocations, randomized page table and a lot of extra cache misses - all the usual causes of generic performance hits. Guess which process that would be.

Copying entire serializable data in one go (without gathering it over entire memory) between game updates may be one of the most performant ways to solve this problem, but would require tricky memory allocation.

On the other note, Factorio has pretty nice deterministic game mechanics, is it utilized anywhere besides saves? For example, to defer (or join several into one) updates on anything that is outside of (everyone's) screen, for example?

N35t0r
Burner Inserter
Burner Inserter
Posts: 15
Joined: Sat Aug 23, 2014 9:29 pm
Contact:

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by N35t0r »

bobingabout wrote: ↑
Mon Nov 26, 2018 9:19 am
Also it would seem the diagonal pole rotations are backwards.
This. Literally unplayable ;-)

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 5206
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by eradicator »

bobingabout wrote: ↑
Mon Nov 26, 2018 9:19 am
Rseding91 wrote: ↑
Fri Nov 23, 2018 7:19 pm
Determinism.
Don't get me wrong, I don't think these "annoying" doodads should be removed, they add character, I just don't see why they need to be synced in the name of determinism.
Just to put this into perspective: Suddenly you are the one asking to remove engine features that other modders might be using to hack around limitations. Also i personally am quite fond of the idea that i see the same things that other people do. There's a certain mission in Dead Space 3 coop where one person sees something completely different, and i can tell you it doesn't feel that great when the other person thinks you're suddenly insane ;). It was funny afterwards, but i wouldn't want that as the default state.
Author of: Belt Planner, Hand Crank Generator, Screenshot Maker, /sudo and more.
Mod support languages: ζ—₯本θͺž, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.

posila
Factorio Staff
Factorio Staff
Posts: 5200
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: Friday Facts #270 - HR Substation & Save/Load overview

Post by posila »

ske wrote: ↑
Sun Nov 25, 2018 7:41 pm
:o I hope I didn't give you any new ideas. I mean I think it could probably be done. What the operating system is doing during COW is not much more than a simple memory copy. If you do that manually you should be more efficient. You'd just have to use an allocator for everything relevant for the game state and then just copy all memory areas allocated by that allocator. That's a few memory areas. Sure you'd need to fix the pointers but that could be done by some lookup-tables or a simple offset if you only have one container. Sure, you'd need to apply that offset to all internal pointers but that could be covered by using templates that optimize away such an offset for the normal operation. After copying you'd only need to serialize the whole package and discard it.
Yes, it could be done, in unreasonable amount of time :). I mean, we are not lacking ideas, we are lacking time dilation device; or time travel device to send kovarex message to 2012 about how big EnergyCraft will get and what things he should plan for better ... and that he should really make it 3D (like Starcraft 2, not first-person) :D

Post Reply

Return to β€œNews”