Page 1 of 1

Factorio Services Outage

Posted: Wed Nov 21, 2018 6:33 pm
by HanziQ
We're currently experiencing DNS issues on the factorio.com domain, which is affecting the availability of official Factorio websites and service, like the forums, wiki, the lua documentation, the mod portal or the matching server. At this time, we're expecting everything to resolve itself eventually, with the worst case time of 20 hours from the posting of this message. We cannot unfortunately do anything to improve this timeframe. To temporarily combat this, you can attempt to set your DNS servers to Google (8.8.8.8, 8.8.4.4) or Cloudflare (1.1.1.1, 1.0.0.1). Please only do this if you understand what it means.

Re: Factorio Services Outage

Posted: Thu Nov 22, 2018 5:05 am
by Tekky
Thanks for providing this information. I am currently only able to access the forums after editing my hosts file, thereby manually providing the IP addresses for www.factorio.com and forums.factorio.com.

I wonder what the problem is. As far as I can tell, all 4 nameservers for the domain factorio.com are providing valid information, yet my local ISP's nameservers are not providing any information for anything inside the factorio.com domain.

Is my guess correct that this is a TTL (time to live) issue?

Do my local ISP's nameservers have an out of date NS record stored in their cache? I guess not, because querying them for the NS record also fails. Maybe it is an issue of negative caching?

Re: Factorio Services Outage

Posted: Thu Nov 22, 2018 5:11 am
by Jap2.0
How are you expecting people to see this if they can't get to *.factorio.com?

Otherwise, I can confirm that changing DNS works (and I've been meaning to do that for quite some time anyway), so thank you for the update on this.

Re: Factorio Services Outage

Posted: Thu Nov 22, 2018 5:35 am
by Nova
I suggest to make a tweet about these problems in the future. Twitter would be the first place to search for informations about offline websites for me. ^^

Re: Factorio Services Outage

Posted: Thu Nov 22, 2018 5:48 am
by Tekky
Jap2.0 wrote:
Thu Nov 22, 2018 5:11 am
How are you expecting people to see this if they can't get to *.factorio.com?
By manually querying the authoritative nameservers and adding the returned IP addresses to the hosts file?

Well, it worked with me. :)

Re: Factorio Services Outage

Posted: Thu Nov 22, 2018 9:41 am
by HanziQ
Jap2.0 wrote:
Thu Nov 22, 2018 5:11 am
How are you expecting people to see this if they can't get to *.factorio.com?
Judging by the traffic on our servers and Heroku apps, not many people were affected, or for example, they might be able to visit the forums, but not the mod portal.
Tekky wrote:
Thu Nov 22, 2018 5:05 am
I wonder what the problem is. As far as I can tell, all 4 nameservers for the domain factorio.com are providing valid information, yet my local ISP's nameservers are not providing any information for anything inside the factorio.com domain.

Is my guess correct that this is a TTL (time to live) issue?

Do my local ISP's nameservers have an out of date NS record stored in their cache? I guess not, because querying them for the NS record also fails. Maybe it is an issue of negative caching?
It's an issue with DNSSEC and its very long (24h) TTLs.

Re: Factorio Services Outage

Posted: Thu Nov 22, 2018 9:53 am
by BenSeidel
HanziQ wrote:
Thu Nov 22, 2018 9:41 am
It's an issue with DNSSEC and its very long (24h) TTLs.
Does anyone else remember when 48 or 72 hours were the norm? I mean, 24 hours was considered "average" at one time. I have had to keep old mail servers up for a week just to ensure that bad DNS records are flushed during a server move. Are you saying that it's an issue with their standard practices?

Re: Factorio Services Outage

Posted: Thu Nov 22, 2018 9:59 am
by HanziQ
BenSeidel wrote:
Thu Nov 22, 2018 9:53 am
HanziQ wrote:
Thu Nov 22, 2018 9:41 am
It's an issue with DNSSEC and its very long (24h) TTLs.
Does anyone else remember when 48 or 72 hours were the norm? I mean, 24 hours was considered "average" at one time. I have had to keep old mail servers up for a week just to ensure that bad DNS records are flushed during a server move. Are you saying that it's an issue with their standard practices?
I'm not sure, I would've expected disabling DNSSEC where your DNS host is also the domain registrar to go smoothly, so the registry record is removed, so everyone stops checking the signature and remove the DS records like a day later, or something.

Re: Factorio Services Outage

Posted: Thu Nov 22, 2018 10:14 am
by BenSeidel
HanziQ wrote:
Thu Nov 22, 2018 9:59 am
I'm not sure, I would've expected disabling DNSSEC where your DNS host is also the domain registrar to go smoothly, so the registry record is removed, so everyone stops checking the signature and remove the DS records like a day later, or something.
Agreed, but always assume that someone out there is using the worst-case settings, the ones that make you do the most amount of work.
Monitor your services and only remove your sever/routing off the old IP once traffic has stopped. Otherwise you end up in this situation.

I'm talking from experience... I've been chewed out for other peoples bad settings, even going so far as having to tell the other party how to clear their DNS cache. Go Team!

Re: Factorio Services Outage

Posted: Thu Nov 22, 2018 10:25 am
by HanziQ
The problem is, that everything is still available on the old versions of any records that changed. The forums A record didn't even change, the whole thing is just some conflict while disabling DNSSEC.

Re: Factorio Services Outage

Posted: Thu Nov 22, 2018 10:42 am
by BenSeidel
So I take it you disabled both the DNSSEC at the registrar level and at the nameserver level at the same time then?

Re: Factorio Services Outage

Posted: Thu Nov 22, 2018 10:50 am
by HanziQ
BenSeidel wrote:
Thu Nov 22, 2018 10:42 am
So I take it you disabled both the DNSSEC at the registrar level and at the nameserver level at the same time then?
Yes, that was the only option available to me from DNSimple, a single button.

Re: Factorio Services Outage

Posted: Thu Nov 22, 2018 10:52 am
by BenSeidel
I sympathise with you.

Re: Factorio Services Outage

Posted: Fri Nov 23, 2018 8:16 am
by featherwinglove
Nova wrote:
Thu Nov 22, 2018 5:35 am
I suggest to make a tweet about these problems in the future. Twitter would be the first place to search for informations about offline websites for me. ^^
Not for me. I have been IP banned from Twitter and haven't been able to use it properly in over three years. This predates the conservative purge of recent months so I guess they really hate me. (OTOH, my Twitter feed hasn't been deleted, so you can go see what happened.)