Friday Facts #415 - Fix, Improve, Optimize

[gGeorg]

Lots of times I'll place a blueprint down next to me, and it takes a while for my personal bots to realize they're the closest. Hopefully this fixes the issue, but if it doesn't, could you set robots in personal equipment grids (i.e., Players, Spiders) to be checked first? if i place a blueprint next to me, I want my personal bots to place it, and place it immediately. But if i place a giant solar farm 1,000 kilometers away, i don't care if there's a 10 second delay until the robots start to place it.

Non- systemic solution, but improves the situation where it hurts most.
I support this,
even when all robots request system is slowed down a bit it worth it. From player point off view, player is the centre of the world, therefore all his needs are the top priority.

[jamaicancastle]

could you set robots in personal equipment grids (i.e., Players, Spiders) to be checked first?

My understanding is that the control flow is "jobs look for bots" rather than "bots look for jobs", so it's not really possible to prioritize a particular set of bots.

However, what's changing in 2.0 is that jobs will be able to judge whether it's faster to request an idle bot from far away, or wait for a close-by bot that's busy to finish its previous tasks. So if you have some bots that are close by (in inventory, or maybe in a roboport that happens to be nearby) it will give those second, third, etc. jobs before calling over more distant bots that will take longer.

[TheoMarque]

Hmm, can you change a method why save game is send from server to client? Tried many times, speed up is slow and lasts about 8 MB/s. Tested many situations, network cards, px, os, servers, platforms and local and via internet. As save games starts to be larger and larger (my last nodded save game was about 500MB /s) loading time before catching up extends due slow download speed (catching up is fast). Probably, a problem is in the logic about servig chunks of zipfile or udp transmission. Some test results I was posted on the forums.
Anyway, when game is dedynced and server dump save game, is not checking for free space to do it and can max out space on server causing corruption. I think, server need to notify or something else to predict space requirements before first save game will be saved. For example, if we set 10 aitosaves history and last save game is about 100 MB, server need to check space for next 11 saves and one for dedynced report.

[gGeorg]

could you set robots in personal equipment grids (i.e., Players, Spiders) to be checked first?

My understanding is that the control flow is "jobs look for bots" rather than "bots look for jobs", so it's not really possible to prioritize a particular set of bots.

Even if you are right. "jobs look for bots" - then each job always check the status of bots in the personal roboport first, then goes into queue of jobs for full searching.

[mrudat]

I've got to wonder if there would be a win to be had to classify build orders for a given roboport network by item; that way, if you don't have enough items/robots, you only need to consult the list as new items and idle robots become available, but the rest of the time you don't need to consult the list at all.

Also, you can consult the list lengths to generate accurate-ish counts for alerts.


For huge saves, it would be nifty if the delta from the last save could be sent instead of having to download the entire save, given that we know the last time any given player was last connected; perhaps a utility that builds and maintains a set of delta files between saves allowing any player to download a set of deltas to bring their local save up-to-date hopefully using only a fraction of the required bandwidth.

[Houdini111]

it's really frustrating to see straight-up bugfixes that sound like they would easily apply to factorio v1, being limited to v2

That's assuming it doesn't conflict with any v2 Specific changes. There's a decent chance that it does, and if it does then they'd have to make it in v1 and then port the changes to v2 (deal with merge conflicts). Which might lead to them having to write the changes twice. And that's assuming that the change would even be possible in v1. It might depend on things only added in v2.
All that to say, we don't know that they would easily apply to v1.

[bobucles]

For huge saves, it would be nifty if the delta from the last save could be sent instead of having to download the entire save, given that we know the last time any given player was last connected; perhaps a utility that builds and maintains a set of delta files between saves allowing any player to download a set of deltas to bring their local save up-to-date hopefully using only a fraction of the required bandwidth.

Delta saving would demand that the server already has the exact same save file that the other computer has when joining in. In effect, the host would need to store a save file for every player on the server, in the worst case scenario. Then it can take the old file, compare to a new file, take the delta, send it, then give the new up to date delta so they can catch up. It would certainly be an unpleasant experience and place a lot of burden on the host to make all these shortcuts available to the new player.

Securing an efficient delta sounds painful. I'm sure the wheel has been invented somewhere, but the tasks for making the delta small become difficult very quickly. Looking for data that has been shifted, new data bits inserted, dealing with compression, stuff like that... that's a project all its own.

When a new player joins, their main limiting resource is bandwidth. The bandwidth comes from the server. Um, what if there was a more distributed system? Anyone familiar with torrents knows that it tries to get bandwidth from multiple sources. In this case, the other sources would be other players on the server. Every active player has the same save file(in theory), after all. There are more than enough digital tools in the world to ensure data accuracy between strangers.

Professional hosts probably won't care. Pro servers tend to be monsters with huge connections. Casual hosts could really benefit though. They wouldn't have the big data streams, but half a dozen players would add a lot of upload speed pretty quickly.

[Syriusz]

Being able to reproduce bugs locally is I guess around 90-95% of the time spent fixing reported bugs.

As fellow developer I feel this so much. I cannot even discuss it with players, and we don't get logs or saves since I work for a studio that has a person to answer on forums, but they never ask for details or files. Sometimes when I am assigned to fixing live game, I end up a day with 0 fixes or even 0 reproductions and this feels bad.

[meganothing]

Being able to reproduce bugs locally is I guess around 90-95% of the time spent fixing reported bugs.

As fellow developer I feel this so much. I cannot even discuss it with players, and we don't get logs or saves since I work for a studio that has a person to answer on forums, but they never ask for details or files. Sometimes when I am assigned to fixing live game, I end up a day with 0 fixes or even 0 reproductions and this feels bad.

And you can't simply talk with this person that answers the forum and tell him/her what you need?

[hukl]

Could we maybe have Robo Districts to separate bots and not have 20000 bots in one gigantic network. Right now we have to separate the networks by physically removing Roboports which leads to other annoying issues like taking extra care where you paste down blueprints with Roboports. A simple UI to allow the user to mark districts in the mini map would be great. I wouldn't mind if you'd add something to assign bots to districts but I can live with the manual bot balancing as long as I wouldn't have to keep the networks separated.

That being said - always great to hear about your optimization efforts

[mrudat]

For huge saves, it would be nifty if the delta from the last save could be sent instead of having to download the entire save, given that we know the last time any given player was last connected; perhaps a utility that builds and maintains a set of delta files between saves allowing any player to download a set of deltas to bring their local save up-to-date hopefully using only a fraction of the required bandwidth.

Delta saving would demand that the server already has the exact same save file that the other computer has when joining in. In effect, the host would need to store a save file for every player on the server, in the worst case scenario. Then it can take the old file, compare to a new file, take the delta, send it, then give the new up to date delta so they can catch up. It would certainly be an unpleasant experience and place a lot of burden on the host to make all these shortcuts available to the new player.

Securing an efficient delta sounds painful. I'm sure the wheel has been invented somewhere, but the tasks for making the delta small become difficult very quickly. Looking for data that has been shifted, new data bits inserted, dealing with compression, stuff like that... that's a project all its own.

There are many delta algorithms, rsync and diff/patch for example, most of which can create a file containing "how to turn file 1 into file 2"; For factorio saves you can't use rsync/xdelta/etc. directly as the save itself is compressed, but it is probably feasible to delta the uncompressed data and create a compressed delta file that can recreate a valid save. It might be necessary to change the order in which things are serialised to achieve the best delta compression, though.

You don't need to retain old saves to produce deltas (you just need save n and save n-1), though it is more efficient to send the delta from 1 to 42 instead of the deltas from 1 to 2, 2 to 3, ..., 41 to 42; the increased bandwidth efficiency is probably not worth the extra space and cpu time on the server.


As for deciding which deltas to send and how many to keep? That's easy enough; "give me every delta since <tick from the save I have>" should work, and if bandwidth is the limiting factor, you delete the oldest delta when the total size of the delta files exceeds the size of the latest save (as downloading the save would be faster).


Doing something akin to a torrent, where you ask the connected clients and the server to please give you a copy of the last save (or the deltas to update your save) does sound like a useful idea; as far as I understand it, doing that for a big save doesn't work well, as your target state keeps changing, but a set of delta files is static-ish (you keep adding new deltas and deleting old ones, but the individual deltas don't change) so copying them from other clients should work just fine.

Hmm. It's possible, if the server is heavily bandwidth-constrained, that holding onto deltas for an extended period might be worthwhile; downloading more data in total from other clients to update your save may be faster than everyone downloading the latest save, but I have no idea where the break-even point might be.


It should be relatively trivial to test how well the various delta algorithms work; grab a big multiplayer save, load it in single-player and wait to generate a set of autosaves, then use those saves to test how small a delta file can be created by rsync/xdelta/etc., and that you can reconstruct a valid save using an older save and the delta file.

[Tec]

Why does the binary on the picture say: G U L E

[tanthus]

That post takes me back to my game developer days (90s). Ahh the strange and weird bugs, the optimizations, the bad poorly written bug reports (thank God for competent testers!). I, ALMOST, miss it I might need to build some mods when 2.0 comes out just for kicks.

[bobucles]

It should be relatively trivial to test how well the various delta algorithms work; grab a big multiplayer save, load it in single-player and wait to generate a set of autosaves, then use those saves to test how small a delta file can be created by rsync/xdelta/etc., and that you can reconstruct a valid save using an older save and the delta file.

The current system is the baseline, "give them the last major save, then update them to the current events". No point in doing something worse than that. The point was in regards to updating a returning player who already has some save data.

The only way to know what data they have is for the host to have a running record of save data on its own. The complexity of such a system builds up fairly quick. How far back does the server track? When does it give up and a say a fresh download is better? How much RAM and hard drive space is the host willing to spend to build up these deltas? Ultimately the host is putting forth a lot of extra effort, to trim down the total data update needed from a returning player.

And it only helps returning players. New players are stuck with a full download, all the same. They don't have a previous file to delta. In fact new players seem to suffer more, because they need to download more stuff. They need the full obsolete file full of garbage, all the deltas to patch it, then the final update to connect. Oh, but maybe the server should do this or that to help various edge cases? It's getting pretty complicated, pretty fast.

The scale of the solution should probably match the scale of the problem. A sever with piles of excess resources running huge delta databases? You know what, I bet that server already has a beefy connection. It probably won't have bandwidth issues and doesn't need help with it. So why make a big server solution, when big servers don't have a problem? A server that lacks bandwidth on the other hand, is likely lacking on other system resources as well. The small computer solution, is asking other computers to help out.

[Jan5366x]

not all heroes wear capes

nice work as usual!

[jjcf89]

it's really frustrating to see straight-up bugfixes that sound like they would easily apply to factorio v1, being limited to v2

v2 is free right? It's only the expansion that will cost money, so everyone should get these improvements, if I understood correctly.

[mrudat]

It should be relatively trivial to test how well the various delta algorithms work; grab a big multiplayer save, load it in single-player and wait to generate a set of autosaves, then use those saves to test how small a delta file can be created by rsync/xdelta/etc., and that you can reconstruct a valid save using an older save and the delta file.

The current system is the baseline, "give them the last major save, then update them to the current events". No point in doing something worse than that. The point was in regards to updating a returning player who already has some save data.

The only way to know what data they have is for the host to have a running record of save data on its own.

You name the delta file for the tick of the old save file, so the client can ask for every delta since the tick of the file it has by the server listing the delta files in ascending order. If the server no longer has the delta file for that specific tick, your save is too old, and you need a fresh one.

How far back does the server track? When does it give up and a say a fresh download is better?

You throw away oldest delta when the total size of the deltas exceeds the size of the last save, as that's the point where using deltas uses more bandwidth than downloading a fresh save.

How much RAM and hard drive space is the host willing to spend to build up these deltas?

Not a huge amount, the read/write buffers for the delta library, a list of the delta files and their sizes, and approximately enough disk space to hold one more save file.

Ultimately the host is putting forth a lot of extra effort, to trim down the total data update needed from a returning player.

I believe the extra effort expended is within an order of magnitude of the cost of compressing the save file, rsync is a relatively simple algorithm in terms of CPU cost and memory consumption. If the save isn't well structured for delta compression, xdelta is more expensive, but will produce a smaller delta.

And it only helps returning players. New players are stuck with a full download, all the same. They don't have a previous file to delta. In fact new players seem to suffer more, because they need to download more stuff. They need the full obsolete file full of garbage, all the deltas to patch it, then the final update to connect.

I figure for new players, you download the latest full save (you now have a recent local save and are logically now a returning player), you might then download a delta if a new save happened while you were downloading the full save because that's almost certainly faster to download and apply than replaying, and then you replay until you're in sync, same as usual.

[Zavian]

Even if you are right. "jobs look for bots" - then each job always check the status of bots in the personal roboport first, then goes into queue of jobs for full searching.

My understanding is that it looks for the closest unassigned bots first. Those are likely to be your personal construction bots. But if you have 50 personal construction robots and place a blueprint that needs 52 robots, then 2 jobs are likely to be assigned to bots that your personal construction bots. That should be improved by the fixes in FFF-374 (https://factorio.com/blog/post/fff-374).

[Zavian]

For huge saves, it would be nifty if the delta from the last save could be sent instead of having to download the entire save, given that we know the last time any given player was last connected; perhaps a utility that builds and maintains a set of delta files between saves allowing any player to download a set of deltas to bring their local save up-to-date hopefully using only a fraction of the required bandwidth.

The delta between 2 saves is likely to be huge, even if the player was last connected 5 minutes ago. Every working assembler will be at a different fraction of it next item completed, and have a different internal inventory of items, every inserter will be at a different point it its swing and might have different hand items. All of the items on every belt are likely to have moved. (In general, the only time the generated delta would small, is if you only send the player orders, and the client had to play catch up. That might work for the case of a client that last connected 5 minutes ago, but could take hours for a client that was last connected 8 hours ago).

Differential or delta saves work well where 90+% of the data is static data that does not change between saves, but that isn't the case with Factorio.

[Yeol]

Ah, multithreading. One of the three events of the asynchronous programming triathlon. The other two being GPU programming and distributed systems. Out of curiosity, what was the fix? I'm guessing enforcing an ordering on the generated chunks, since if it was a missing synchronization primitive you'd probably get way weirder bugs even in single player.