Friday Facts #231 - Belt compression & Crash log uploading

[Ghoulish]

I'm glad you have chosen not to change the uploading of crash logs, it's without doubt the best choice for the game.

[Mobius1]

About the AMD Ryzen debug process:
Its an AM4 socket processor, used with DDR4 memories that usually operates on OC mode.
I personally think that the :PrepareRow procedure causes the crash because factorio is deterministic and the Ryzen Processor has its "40% gain" on using QuickSort on the job queues that goes into the processor, which will be forcefully converted to 4 threads to do the process "faster" but that makes it undeterministic by nature since you don't control how the processor will split the job into the 4 threads and also you don't control which thread will finish earlier to give the result first.
One approach that I think could be used to test this sort of issue would be doing a thread treatment on the :PrepareRow procedure, like a debug integrity check on the data, since factorio checks if the data before and after the job are the same state, the randomness of the issues you guys are receiving on the crash report are desyncs caused by the processor itself, so you would need to assign steps to the debugger especially for the Ryzen computers "{$IFDEF RYZEN}" and on every procedural step of the :PrepareRow procedure, even during loops, you guys would do a "state check" on the processed data, with that you'll see how the processor would scramble the job data that it receives, resulting on different job completion times that causes the "processor desync" which causes the crash.

Imagine those situations:
- Ryzen receives factorio::onLoad job
- Ryzen receives windows media player::playnext job
- Ryzen receives google chrome::onUpdate job
internally it uses QuickSort algorithm to get all jobs and split them into 4 threads per core, so for a quadcore Ryzen, it would be 16 threads:
- Ryzen receives thread#1 factorio::onLoad #1
- Ryzen receives thread#2 windows media player::playNext #1
- Ryzen receives thread#3 factorio::onLoad #2
- Ryzen receives thread#4 google chrome::onUpdate #1
- Ryzen receives thread#5 factorio::onLoad #3
- Ryzen receives thread#6 google chrome::onUpdate #2
- Ryzen receives thread#7 google chrome::onUpdate #3
- Ryzen receives thread#8 google chrome::onUpdate #4
- Ryzen receives thread#9 google chrome::onUpdate #5
- Ryzen receives thread#10 windows media player::playNext #2
- Ryzen receives thread#11 google chrome::onUpdate #6
- Ryzen receives thread#12 google chrome::onUpdate #7
- Ryzen receives thread#13 google chrome::onUpdate #8
- Ryzen receives thread#14 google chrome::onUpdate #9
- Ryzen receives thread#15 factorio::onLoad #4
- Ryzen receives thread#16 factorio::onLoad #5
- Ryzen finishes thread#5 factorio::onLoad #3 <----- on a loading part of the game, factorio isn't deterministic, but you see here that the processing of jobs aren't on a deterministic way which causes the desync on functions that needs that certainty.
The above process isn't 100% accurate its just a tabletest usecase, hope it helps.

[bobingabout]

can you tell us which Ryzen CPU is affected or when it is happening (what are players doing)? So it might be tested by players?
Have Ryzen 7 1700X myself on 2 PC ^^

We asked that in the discord chat. the answer was "The ones made by AMD". It might have been a Joke. That or they don't know exactly which models yet.

[Mimos]

Do you have an affected Ryzen processor yourself for running tests on?

Nope, we were thinking of possibly getting one. But the issue is still very rare and we will most probably not be able to trigger the crash on our computer.
Last time we got and AMD computer with weird hardware we did a few tests and now it's mostly collecting dust.

There was talk about compiler crashes on earlier versions of the processors. AMD offers replacement for those earlier processors that are affected by the compiler crashes. Threadripper/EPYCs were already fixed for that particular crash type at release. It might be some weird caching problem that occurs in conjunction with tight loops. Do you check and include the CPU stepping in the crash logs?

can you tell us which Ryzen CPU is affected or when it is happening (what are players doing)? So it might be tested by players?
Have Ryzen 7 1700X myself on 2 PC ^^

We asked that in the discord chat. the answer was "The ones made by AMD". It might have been a Joke. That or they don't know exactly which models yet.

I think I even read somewhere that they didn't change the stepping for this particular fix and you have to look at the production date which is etched into the heatspreader (in a slightly obfuscated form). Some dates are known to be ok, some are known to be unaffected. I'm not sure if the exact date starting with wich the CPUs are ok is known publicly. So I guess it will be quite hard to classify the CPUs automatically for checking if this bug might be caused by the same hardware issue.

Edit: The article that explains this was already linked:

Here's a crazy idea: try making a release with optimization settings changed for that one function and see if the frequency of crash reports change. There's a chance you found an erratum, maybe another instance of this one? https://www.phoronix.com/scan.php?page= ... ixed&num=1

[bobucles]

Code: Select all

generator will now produce pollution if emissions is enabled on fuel source

No! What will I do without my billowing green clouds of nuclear steam power???

I do like the idea of processing fuel to reduce boiler pollution. It seems like a good incentive to process everything.

[TigBits]

All the sperging out over uploading crash logs is less about privacy and more about piracy. Thieves don't like it when their stolen goods phone home.

Aside from that, anyone whining about privacy and sill using chrome/google/facebook on a daily basis is a hypocrite and an idiot.

[Ubertwink]

can you tell us which Ryzen CPU is affected or when it is happening (what are players doing)? So it might be tested by players?
Have Ryzen 7 1700X myself on 2 PC ^^

We asked that in the discord chat. the answer was "The ones made by AMD". It might have been a Joke. That or they don't know exactly which models yet.

If it's Ryzen specific, it will happen on any Ryzen cpu, since they all are the same chip with same architecture.

I believe this crash is an indication of bad memory overclocking. Ryzen overclockers know that stress tests can't find 100% of system instabilities and games are the best way to do that. Source: Ryzen user myself.

Before you ask: selecting an XMP profile in bios is considered overclocking too. Most XMPs are designed for Intel and AM4 has different DDR4 implementation.

For devs: if you can, try collecting memory speed and primary timings as well, although I'm not sure how anti virus programs would behave if Factorio suddenly starts to poll SMBUS.

[Nova]

Do the right thing and be a role model for everyone by asking the player after installing / updating the game if they want to send crash reports.

Thank you for finally addressing privacy. I have been dealing with log files myself and I can't help but wonder why people still feel unsafe about it. OR/AND worry about those who do not even care.

I still recommend not even making this an option though.

Why we feel unsafe about it? Because there are enough examples of people doing bad stuff with uploaded data. Even trusting someone doesn't work - most evil people would say "trust me", even if you can't trust them. The right way to adress this is making it as transparent as possible. Just starting to upload stuff doesn't do that, sorry.

Apart from the extra implementation time, players will still often click "Don't send", either because that's what they have been doing for many years or because they just want to quickly get back in the game or because they feel that the crash was somehow their fault.

Um, in that case, you just don't get any data, period. "We don't give people the option because they would choose the wrong answer." is a terrible way of handling things.
The real problem I see is that you upload data without asking the player first. A person who doesn't read the whole changelog and doesn't scan the options for something like this won't notice that you upload his data. Ask the player at the beginning of the game if he wants to upload anonymous crash reports. You can do it with a checked checkbox - users would have to deactivate it to disable the data upload. That would in theory still be opt-out, but people are forced to read it - if they skip it, their problem.

Every data is valuable. Data can be stolen. The only data you can't steal is data that doesn't exist. We don't know how secure the factorio servers are. We don't know if there will ever be a bug / security hole that breaches all of your security systems. Don't force people to have trust into your security. I have no problem with uploading crash reports to your servers, but I have a problem if I'm forced to do that. You fortunately give us the option, that's already a big deal, thank you. Even better would be asking us before you do it.

You could say "but they are anonymous!" - well, you don't do it yet: "In the next version we even sanitized the log a bit so it does not contain anything that might be seen as a privacy breach, such as IP addresses." - that's a pretty weird way of doing sensitive stuff. There are thousands of Factorio players, thousands of crashes, and you uploaded their data without removing (all) sensitive informations? And then we should trust you? That's a stretch.

Do the right thing and be a role model for everyone by asking the player after installing / updating the game if they want to send crash reports.
I always liked the Factorio devs for doing the right thing (providing a demo of the game, ...) and trying to be honest with the player (cost of 20 Euro for the game and not 19.99, ...). Please rething this crash report stuff so the player base can still believe in your awesomeness in every way.

[brunzenstein]

Please rething this crash report stuff so the player base can still believe in your awesomeness in every way.

Calm down.
We (WUBE is located in the EU) are not only in the USA present.
European law regarding user data is quite strict now and will be robust updated March 2018 and enforced ruthless europe wide by the EU comission.
Companies will face extinction (the fines are extremely high and base on the individual companies market capitalisation or value - in worst case a better share of market value may be the fine) when overstepping EU rules.

[TheVeteraNoob]

How's about this idea to put on the pile.

When factorio crashes it has the whole "factorio has crashed" kinda popup we all know and love. Then it has a check box saying automatically send crash reports. If the box is checked when you hit ok or close or whatever it will send. But unchecking will change the setting and no longer update.

You might say other companies do worse things and I have no problem with the intentions but as one user has mentioned. All data has value and collecting data without consent that nobody else is collecting attaches a very tempting value to those who might try and obtain it

[traycer]

Wube will be subject to the GDPR. However they decide to implement crash log reporting, if it's good enough for that, it's good enough for me. I looked through some saved crash logs and the only thing that might even be remotely considered PII is my Windows username embedded in some pathnames. Not even an IP address (maybe only in multiplayer?). Sending crash logs by default seems reasonable to me in this case, given the benefits mentioned by Twinsen. The player is notified of this via FFF and the changelog, and is given the opportunity to disable it at any time. Seems more than fair to me for information that has little to no value to anyone other than Wube.

[Paul17041993]

@Klonan You should try to collect at least the motherboard brand as well, preferably the chipset info too, as there's known issues with AM4 gigabyte and MSI boards that could be contributing to the render crash...

[Programmdude]

All the sperging out over uploading crash logs is less about privacy and more about piracy. Thieves don't like it when their stolen goods phone home.

I own windows 10 and factorio, I hate it that windows 10 has mandatory tracking, because I don't trust microsoft. However I don't mind factorio uploading crash logs for three reasons, I trust them as a company, crash logs dont monitor all my PC usage, just when I am playing factorio, and lastly, they are in the EU, which has much saner privacy laws.

[Tysheth]

[...] ask for the opt-in / opt-out once in the main menu for the people who haven't made a choice yet. Show it in a popup over the main menu [...]

This is what I intended to suggest. At the very least, pull the option out of the settings menu and put it in a check-mark on the main screen. It feels very shady for the phone-home opt-out to be hidden away.

I have no problem with uploading crash reports to your servers, but I have a problem if I'm forced to do that. You fortunately give us the option, that's already a big deal, thank you. Even better would be asking us before you do it.

You could say "but they are anonymous!" - well, you don't do it yet: "In the next version we even sanitized the log a bit so it does not contain anything that might be seen as a privacy breach, such as IP addresses." - that's a pretty weird way of doing sensitive stuff. There are thousands of Factorio players, thousands of crashes, and you uploaded their data without removing (all) sensitive informations? And then we should trust you? That's a stretch.
[...]
I always liked the Factorio devs for doing the right thing (providing a demo of the game, ...) and trying to be honest with the player (cost of 20 Euro for the game and not 19.99, ...). Please rething this crash report stuff so the player base can still believe in your awesomeness in every way.

I am seconding everything here.

You can't expect all customers to comb through the settings as the first thing they do after starting up the game. Before you chose to take identifying information without explicitly informing customers, I held you guys in my mind as the ideal for-profit developers.

[Light]

I'll add that I use the Threadripper 1950X on a Gigabyte board with all hardware components overclocked to their limits, yet fortunately haven't experienced this bug. It's certainly something I'll report on if I notice from this point onward, but most bugs are usually mod related.

I own windows 10 and factorio, I hate it that windows 10 has mandatory tracking, because I don't trust microsoft.

It's not mandatory, you just didn't bother to look into your available options to turn it off. Options have always existed to disable telemetry since WinXP and haven't gotten any harder to disable since then. If it truly bothered you then you would have looked into disabling it rather than bitching about it like you're tied down to it.

Any argument stating that "Well if X company does it then it's ok for us too!" is a complete fallacy. I'll say it again, there's almost always a way to disable privacy invading methods if you're so inclined to look into it. However, most people will take the easy road and bitch rather than put in that bit of effort to correct the problem and move on.

Since Factorio has the opt-out option clearly available in game and it's listed in their policies, there's little reason I can find to fault them for it. It would be nice if they went for the pop-up asking to send reports, but they aren't being shady about uploading the data regardless of what you want.

Improved transparency is also a nice thing to hear Wube wants to take seriously. Discord's privacy policy clearly states that they do not respect the privacy of the users and gladly do what they please with all the data they store on their servers, yet people don't bother to read those policies and assume they're all fine. Issues such as this are the fault of the user and not the company, which is often the case in just about every situation. That's not shady behavior because it's clearly stated in their policies, it's just a dick thing to do. But at least they're being upfront about that.

[wren6991]

In just 2 days Rseding fixed about 12 crashes

Isn't this normal for Rseding

[aober93]

I'd assume, a crash dump with enough hardware information with computer aided parsing to allow for a hardware fingerprint (IE personally identifying) is a privacy concern. It doesnt matter if the company is so small and the intends to be nice or what. Its just a matter of it being what it is.
That may explain the bad messaging you get.

And hurray for the belt compression change. I always thought you are insane, talk about belts being too weak vs bots, and then praising the belt compression issue as a feature!

[Rseding91]

All the sperging out over uploading crash logs is less about privacy and more about piracy. Thieves don't like it when their stolen goods phone home.

Aside from that, anyone whining about privacy and sill using chrome/google/facebook on a daily basis is a hypocrite and an idiot.

It is fun to see crash logs where the person clearly pirated it But I don't really care. If it helps us fix a bug then they're helping us make the game better.

[Ubertwink]

All the sperging out over uploading crash logs is less about privacy and more about piracy. Thieves don't like it when their stolen goods phone home.

Aside from that, anyone whining about privacy and sill using chrome/google/facebook on a daily basis is a hypocrite and an idiot.

It is fun to see crash logs where the person clearly pirated it But I don't really care. If it helps us fix a bug then they're helping us make the game better.

Can you add some sort of DRM protection so that it's impossible to turn off this feature in the pirated version of the game?

[Narnach]

Nice to read about the crash logs and how it helped resolve some bugs. As a developer working on server-side applications I know how incredibly useful it is to have crashes automatically gathered and organized to help you prioritise what bugs to fix. Happy hunting!

Regarding the opt-in behaviour: personally I don't mind, but for the folks that are a bit more wary of it, you could show a popup once on startup that notifies of the default/current setting, and tells people where to disable it. That way people that are fine with sending crash logs (or the majority that does not care) will default to sending you the logs. The few that don't like it will then explicitly have been told about it and know how to disable it.

As to forum bug reports with crash dumps: there's a lot of manual steps involved, in locating the dump files, searching the bug forums to see if an issue has been reported/resolved yet (can take quite some time to ensure I don't post a duplicate), and then posting about it, so most folks will likely just boot the game and try to see if it was a "random error". If it does not happen often (or often enough) then they will most likely forget. I think I reported one or two errors in total, one that prevented the game from starting (so I was rather motivated) and one for a reliable multiplayer crash bug.

Giving people the option to attach a note to the crash dump to indicate what they were doing might help if you find you need more context for some bugs. Maybe tie it to the bug post forum in some way and use the crash dump to check for duplicates / auto-append notes to the thread about a crash dump?