Friday Facts #415 - Fix, Improve, Optimize

[Mithaldu]

it's really frustrating to see straight-up bugfixes that sound like they would easily apply to factorio v1, being limited to v2

v2 is free right? It's only the expansion that will cost money, so everyone should get these improvements, if I understood correctly.

the problem is that a lot of mods will get ported to v2 only very slowly. if ever.

users who enjoy playing with mods whose authors don't have the time could benefit from improvements such as fixes to the logi bot algorithms, but if the bugfixes are limited to a compatibility-breaking releases, they won't be able to

[mrudat]

the problem is that a lot of mods will get ported to v2 only very slowly. if ever.

users who enjoy playing with mods whose authors don't have the time could benefit from improvements such as fixes to the logi bot algorithms, but if the bugfixes are limited to a compatibility-breaking releases, they won't be able to

Depending on how much the API changes in v2, it might be possible to write a script to update most mods to v2; I've thrown such a script together for a previous breaking change, and it handled most of the mods I was using at the time.

Many mods just needed to have the version bumped.

Edit: Of course, some mods need an actual rewrite, which is beyond a script that just did a search and replacedestroy.

[Hares]

Dear Factorio devs.

Is it possible to make a single-player-only feature (let's be honest, the majority of us play mostly single-player) to allow multi-thread processing of the game state to allow faster updates and more efficient CPU usage? This might be a very nice addition to a larger maps (which are, of course, different planets of Space Age). With that, having a megabase running on 100 UPS (>60!) would be possible!

[Hares]

the problem is that a lot of mods will get ported to v2 only very slowly. if ever.

users who enjoy playing with mods whose authors don't have the time could benefit from improvements such as fixes to the logi bot algorithms, but if the bugfixes are limited to a compatibility-breaking releases, they won't be able to

Depending on how much the API changes in v2, it might be possible to write a script to update most mods to v2; I've thrown such a script together for a previous breaking change, and it handled most of the mods I was using at the time.

Many mods just needed to have the version bumped.

Edit: Of course, some mods need an actual rewrite, which is beyond a script that just did a search and replacedestroy.

I assume that Factorio devs would make some backwards-compatibility features on their side, so even bumping version won't be necessary because, let's be honest, we all use mods from people no longer playing Factorio.

[Svip]

I assume that Factorio devs would make some backwards-compatibility features on their side, so even bumping version won't be necessary because, let's be honest, we all use mods from people no longer playing Factorio.

Considering it has been necessary in the past, it seems weird they would change now. A mod labelled for version 1.0 won't work with 1.1. And indeed, some mods have died that way. Fortunately, some times others come in and pick it up.

Whether the Factorio developers go out of their way to bump certain popular mods manually, I don't know. But it won't be automatic.

[VampireSilence]

Re: Faster construction robot tasks
Why don't you take the "nearest roboport for that chunk" instead of "... for that job"? That way it's a tiny bit inaccurate, but it doesn't really matter a lot and you could cache that roboport for every chunk, updated whenever a new roboport is built. So there wouldn't even be a calculation, rather than a simple lookup.

[SirGouki]

Re: The robot issues.

One of the problems with this I have noticed, is that construction robots seem to just outright ignore the fact that they can carry more than one object, regardless of how much you've researched their storage upgrades. So construction robots will only ever build 1 building, regardless of how many they should actually be capable of building. What's odd, is they only seem to have this problem with buildings, and not the contents, with deconstruction. For example, if you deconstruct an assembler with something in it, and one robot can pick up that entire something (say it has only 3 green chips in it, and their current storage is 4), one construction robot will take all of that item out of the machine, go to drop it off, and another robot will come pick up the machine itself. However, if you're building a solar array, each power pole, solar panel, and accumulator in the blue print tasks a separate robot, regardless of how many objects a robot can carry. Not exactly sure how bad solving this would be, but there could be something that checks "If a robot already has a construction order for X, there are more X available, and it can pick up more, add me to their queue" What's really odd is personal roboport bots do not seem to have this problem with conveyor belts, but still do it to buildings.

[jamaicancastle]

One of the problems with this I have noticed, is that construction robots seem to just outright ignore the fact that they can carry more than one object, regardless of how much you've researched their storage upgrades.

They do, insofar as they'll only take one construction/deconstruction job at a time, regardless of whether they have more capacity. (Except for when building tiles, which was mentioned back in FFF-318.)

Not exactly sure how bad solving this would be, but there could be something that checks "If a robot already has a construction order for X, there are more X available, and it can pick up more, add me to their queue" What's really odd is personal roboport bots do not seem to have this problem with conveyor belts, but still do it to buildings.

Apparently back then they would check for additional construction jobs of the same type and it was pretty hard on TPS if they searched more than the immediately surrounding tiles or for more than 1x1 buildings. Now that there's a queue, it might be possible to do a comparison to jobs in queue relatively cheaply, but I'm not sure. AFAIK the bot searching logic only knows when and where the bot will become idle again, not what its jobs are.

[numzero]

For huge saves, it would be nifty if the delta from the last save could be sent instead of having to download the entire save, given that we know the last time any given player was last connected; perhaps a utility that builds and maintains a set of delta files between saves allowing any player to download a set of deltas to bring their local save up-to-date hopefully using only a fraction of the required bandwidth.

The delta between 2 saves is likely to be huge, even if the player was last connected 5 minutes ago. Every working assembler will be at a different fraction of it next item completed, and have a different internal inventory of items, every inserter will be at a different point it its swing and might have different hand items. All of the items on every belt are likely to have moved. (In general, the only time the generated delta would small, is if you only send the player orders, and the client had to play catch up. That might work for the case of a client that last connected 5 minutes ago, but could take hours for a client that was last connected 8 hours ago).

Differential or delta saves work well where 90+% of the data is static data that does not change between saves, but that isn't the case with Factorio.

From what I've read about saves, the majority of a save is (relatively) static. Decorative entities, trees, tiles and relatively static entities (power poles, idle turrets, walls) outnumber active entities (assemblers, inserters, pipes, ...); removing decoratives can shrink a save significantly, for example.

I believe belts are serialised as they are represented in memory, and runs of items and gaps are updated, not the location of each item. That means a full belt moving at full speed only changes between saves where there are gaps at either end, no matter how long the belt is.

That said, I haven't checked what the actual % change between one save and the next is (yet), and it's entirely possible that, as you say, it's just not worth the effort.

I suspect it would heavily depend on what % of the factory remains idle between saves and how much active combat there is.

I just did a quick check, on some old saves I found (29.0 and 30.6 MiB zips, a few hours apart). xdelta on those (fully uncompressed, 87.2 and 91.3 MiBs respectively) yielded a 3.3 MiB diff (not further compressible with anything I have on hand).

That’s not the full story though. First, fully uncompressing a save isn’t exactly trivial as instead of using zip compression, most files are compressed with zlib and then are put into the zip. That’s weird as that’s what zip would do on its own.

Second, the compression is lower than it could be. Recompressing with gzip -9 resulted in 26.1 and 27.5 MiB archives. And with lzip -9 instead, I got 21.0 and 22.1 MiB respectively which is ~30% less than the original.

Third, the parts are compressed individually. That’s not very space-efficient though it my case, the gains weren’t large, just 1 MiB with lzip and marginal 0.1 MiB with gzip.

[adam_bise]

I assume that Factorio devs would make some backwards-compatibility features on their side, so even bumping version won't be necessary because, let's be honest, we all use mods from people no longer playing Factorio.

Considering it has been necessary in the past, it seems weird they would change now. A mod labelled for version 1.0 won't work with 1.1. And indeed, some mods have died that way. Fortunately, some times others come in and pick it up.

Whether the Factorio developers go out of their way to bump certain popular mods manually, I don't know. But it won't be automatic.

I think you can try updating the version in the mod just to see if it already works. If so, upload it. If not you might have to do a few tweaks, but it isn't an impossible feat and the modding forum is pretty helpful about these things.

[motmontheinternet]

the problem is that a lot of mods will get ported to v2 only very slowly. if ever.

users who enjoy playing with mods whose authors don't have the time could benefit from improvements such as fixes to the logi bot algorithms, but if the bugfixes are limited to a compatibility-breaking releases, they won't be able to

This really isn't Wube's problem. There's a difference between supporting mods and supporting -a- mod. You're asking them to support a mod, really, not modding in general. Individual mods are to be maintained by their maintainer, not Wube. There is no expectation for improvements to be backported.

Sorry for my stupid question but why the user was using Factorio on multiple computers? I mean if it's just a server with multiple players on different machines, there would be desync on all machines ever, no ? Im confused...

There's two reasonable possibilities, I think. One is that he encountered a bug when with his friends and was trying to use his homelab to test the problem. The other is that he was doing something like playing Space Exploration and hadn't gotten to spidertrons yet but still wanted to conveniently exist on every planet he had colonized so he can just play normally without having to travel.

[wild_dog]

So about the latest optimizations to the roboport algorithm.

First off, that is amazing, cudos.

But I'm wondering if there aren't some further optimizations to be had.

Roboport networks are rarely fully stuffed with drones, and big networks are mostly to be able to cover a wider area, rahter than to provide more drone capacity. So is tracking which roboports actually contain any drones, and limiting the search to those, an optimization option? That would change it from O(roboport-count) to O( min( roboport-count, drone-count )).

But the averate use case would likely be much better, due to the drone distribution. Drones tend to clump up in areas with the most activity (recenty), since drones go to the nearest roboport with room available from where they finished their task. So construction drones tend do clump in the last major (de)construction site, or sites with frequent repair tasks such as the walls. And logistics bots tend to clump around the target logistics chests with most throughput.

This can still be a significant savings. I'm willing to bet that in that huge example base, the location of the drones were confined to less than 20% of all roboports.

[wild_dog]

Re: The robot issues.

One of the problems with this I have noticed, is that construction robots seem to just outright ignore the fact that they can carry more than one object, regardless of how much you've researched their storage upgrades. So construction robots will only ever build 1 building, regardless of how many they should actually be capable of building. What's odd, is they only seem to have this problem with buildings, and not the contents, with deconstruction. For example, if you deconstruct an assembler with something in it, and one robot can pick up that entire something (say it has only 3 green chips in it, and their current storage is 4), one construction robot will take all of that item out of the machine, go to drop it off, and another robot will come pick up the machine itself. However, if you're building a solar array, each power pole, solar panel, and accumulator in the blue print tasks a separate robot, regardless of how many objects a robot can carry. Not exactly sure how bad solving this would be, but there could be something that checks "If a robot already has a construction order for X, there are more X available, and it can pick up more, add me to their queue" What's really odd is personal roboport bots do not seem to have this problem with conveyor belts, but still do it to buildings.

One build task that construction drones DO do in multiples, is ground tile placement.

That said, ground tile removal (of adjacent tiles) or wall placement/removal isn't utalizing this.

[Mithaldu]

This really isn't Wube's problem. There's a difference between supporting mods and supporting -a- mod. You're asking them to support a mod, really, not modding in general. Individual mods are to be maintained by their maintainer, not Wube. There is no expectation for improvements to be backported.

There is a lot of things that aren't Wube's problem.

More importantly tho, what is your prescription to me? As far as i can tell right now, all you want me to do is shut the fuck up, which overlooks that without a problem being expressed, chances of it being recognized are low. Or are you telling me to do the work to port them forwards? I already am an open source developer and have contributed to numerous factorio mods and will continue to do so, but my single solitary being's time is not enough to address this problem. Additionally there are also mods that have licenses that forbid this kind of thing.

At the end of the day: There is a problem here. I have expressed it, and given it a chance to be recognized and dismissed or solved. Feel free to provide constructive feedback, if you wish, but otherwise this is entirely up to the dev team.

[FuryoftheStars]

This really isn't Wube's problem. There's a difference between supporting mods and supporting -a- mod. You're asking them to support a mod, really, not modding in general. Individual mods are to be maintained by their maintainer, not Wube. There is no expectation for improvements to be backported.

There is a lot of things that aren't Wube's problem.

More importantly tho, what is your prescription to me? As far as i can tell right now, all you want me to do is shut the fuck up, which overlooks that without a problem being expressed, chances of it being recognized are low. Or are you telling me to do the work to port them forwards? I already am an open source developer and have contributed to numerous factorio mods and will continue to do so, but my single solitary being's time is not enough to address this problem. Additionally there are also mods that have licenses that forbid this kind of thing.

At the end of the day: There is a problem here. I have expressed it, and given it a chance to be recognized and dismissed or solved. Feel free to provide constructive feedback, if you wish, but otherwise this is entirely up to the dev team.

From what I see, including backward compatibility could lead towards game size bloat (as many functions stick around or persist in several forms), may require extra code to translate (either of which could lead to performance drops), and still others may be impossible to keep compatibility for.

Don't get me wrong, I completely understand the desire to keep old mods alive, but to me, I'm not sure it would be worth it.

[motmontheinternet]

There is a lot of things that aren't Wube's problem.

More importantly tho, what is your prescription to me?

I'm explaining to you how things work, which you don't like. No really, don't complain about having limited time while asking Wube to waste theirs.

[Mithaldu]

including backward compatibility could lead towards game size bloat

understandable concern, but in this case simply not applicable, since it would require no code additions as you described whatsoever

don't complain

so your prescription is indeed "shut the fuck up", and i'd invite you to do some self-introspection, and maybe also realize they don't need you to protect them from my posts. they can make their own decisions. (also your wild guesses as to what i, as a professional developer, understand or don't, are hella wrong )

[FuryoftheStars]

including backward compatibility could lead towards game size bloat

understandable concern, but in this case simply not applicable, since it would require no code additions as you described whatsoever

If a function becomes outdated and they'd want to remove normally, the "backward compatibility" options would be to leave it, or create a version of it that translates to the new/replacement function if there is one (which also applies to functions that are merely updated to have different inputs vs being removed/replaced entirely). Code "addition" or not, it will lead to the game being larger than what it would've been otherwise at the very least, and increased performance costs on those mods that don't update if the "backward compatibility" option was to translate from old -> new as there would be that extra code execution to translate it. Depending on the function and its frequency of use, this may have negligible effects, or it could make the mod unusable anyway.

It should also be noted that with maintaining "backward compatibility" comes testing to make sure the functions actually still work (just because they worked before doesn't mean that a change elsewhere later won't break them), and maintaining documentation on them.

Sometimes a mod breaking change brings a modder back to update it. Sometimes it leads to others creating "forks". I'd dare say, too, that mods with licenses preventing even "fork" versions are extremely rare, and if truly abandoned, then they won't be enforced. If someone attempts to enforce it, then that means they're still around and either plan on updating it, or they want it to die. (And honestly, I doubt the licenses actually are enforceable unless the forker is attempting to get money in some way for it.)

[Mithaldu]

If a function becomes outdated and they'd want to remove normally

my complaint isn't even remotely about that though

[Tiavor]

Could we get api access to "found ghost without materials" as an event?