Factorio Services Outage

[HanziQ]

We're currently experiencing DNS issues on the factorio.com domain, which is affecting the availability of official Factorio websites and service, like the forums, wiki, the lua documentation, the mod portal or the matching server. At this time, we're expecting everything to resolve itself eventually, with the worst case time of 20 hours from the posting of this message. We cannot unfortunately do anything to improve this timeframe. To temporarily combat this, you can attempt to set your DNS servers to Google (8.8.8.8, 8.8.4.4) or Cloudflare (1.1.1.1, 1.0.0.1). Please only do this if you understand what it means.

[Tekky]

Thanks for providing this information. I am currently only able to access the forums after editing my hosts file, thereby manually providing the IP addresses for www.factorio.com and forums.factorio.com.

I wonder what the problem is. As far as I can tell, all 4 nameservers for the domain factorio.com are providing valid information, yet my local ISP's nameservers are not providing any information for anything inside the factorio.com domain.

Is my guess correct that this is a TTL (time to live) issue?

Do my local ISP's nameservers have an out of date NS record stored in their cache? I guess not, because querying them for the NS record also fails. Maybe it is an issue of negative caching?

[Jap2.0]

How are you expecting people to see this if they can't get to *.factorio.com?

Otherwise, I can confirm that changing DNS works (and I've been meaning to do that for quite some time anyway), so thank you for the update on this.

[Nova]

I suggest to make a tweet about these problems in the future. Twitter would be the first place to search for informations about offline websites for me. ^^

[Tekky]

How are you expecting people to see this if they can't get to *.factorio.com?

By manually querying the authoritative nameservers and adding the returned IP addresses to the hosts file?

Well, it worked with me.

[HanziQ]

How are you expecting people to see this if they can't get to *.factorio.com?

Judging by the traffic on our servers and Heroku apps, not many people were affected, or for example, they might be able to visit the forums, but not the mod portal.

I wonder what the problem is. As far as I can tell, all 4 nameservers for the domain factorio.com are providing valid information, yet my local ISP's nameservers are not providing any information for anything inside the factorio.com domain.

Is my guess correct that this is a TTL (time to live) issue?

Do my local ISP's nameservers have an out of date NS record stored in their cache? I guess not, because querying them for the NS record also fails. Maybe it is an issue of negative caching?

It's an issue with DNSSEC and its very long (24h) TTLs.

[BenSeidel]

It's an issue with DNSSEC and its very long (24h) TTLs.

Does anyone else remember when 48 or 72 hours were the norm? I mean, 24 hours was considered "average" at one time. I have had to keep old mail servers up for a week just to ensure that bad DNS records are flushed during a server move. Are you saying that it's an issue with their standard practices?

[HanziQ]

It's an issue with DNSSEC and its very long (24h) TTLs.

Does anyone else remember when 48 or 72 hours were the norm? I mean, 24 hours was considered "average" at one time. I have had to keep old mail servers up for a week just to ensure that bad DNS records are flushed during a server move. Are you saying that it's an issue with their standard practices?

I'm not sure, I would've expected disabling DNSSEC where your DNS host is also the domain registrar to go smoothly, so the registry record is removed, so everyone stops checking the signature and remove the DS records like a day later, or something.

[BenSeidel]

I'm not sure, I would've expected disabling DNSSEC where your DNS host is also the domain registrar to go smoothly, so the registry record is removed, so everyone stops checking the signature and remove the DS records like a day later, or something.

Agreed, but always assume that someone out there is using the worst-case settings, the ones that make you do the most amount of work.
Monitor your services and only remove your sever/routing off the old IP once traffic has stopped. Otherwise you end up in this situation.

I'm talking from experience... I've been chewed out for other peoples bad settings, even going so far as having to tell the other party how to clear their DNS cache. Go Team!

[HanziQ]

The problem is, that everything is still available on the old versions of any records that changed. The forums A record didn't even change, the whole thing is just some conflict while disabling DNSSEC.

[BenSeidel]

So I take it you disabled both the DNSSEC at the registrar level and at the nameserver level at the same time then?

[HanziQ]

So I take it you disabled both the DNSSEC at the registrar level and at the nameserver level at the same time then?

Yes, that was the only option available to me from DNSimple, a single button.

[BenSeidel]

I sympathise with you.

[featherwinglove]

I suggest to make a tweet about these problems in the future. Twitter would be the first place to search for informations about offline websites for me. ^^

Not for me. I have been IP banned from Twitter and haven't been able to use it properly in over three years. This predates the conservative purge of recent months so I guess they really hate me. (OTOH, my Twitter feed hasn't been deleted, so you can go see what happened.)