Factorio Services Outage

Regular reports on Factorio development.
Post Reply
User avatar
HanziQ
Factorio Staff
Factorio Staff
Posts: 630
Joined: Fri Mar 27, 2015 7:07 am

Factorio Services Outage

Post by HanziQ » Wed Nov 21, 2018 6:33 pm

We're currently experiencing DNS issues on the factorio.com domain, which is affecting the availability of official Factorio websites and service, like the forums, wiki, the lua documentation, the mod portal or the matching server. At this time, we're expecting everything to resolve itself eventually, with the worst case time of 20 hours from the posting of this message. We cannot unfortunately do anything to improve this timeframe. To temporarily combat this, you can attempt to set your DNS servers to Google (8.8.8.8, 8.8.4.4) or Cloudflare (1.1.1.1, 1.0.0.1). Please only do this if you understand what it means.

Tekky
Filter Inserter
Filter Inserter
Posts: 859
Joined: Sun Jul 31, 2016 10:53 am

Re: Factorio Services Outage

Post by Tekky » Thu Nov 22, 2018 5:05 am

Thanks for providing this information. I am currently only able to access the forums after editing my hosts file, thereby manually providing the IP addresses for www.factorio.com and forums.factorio.com.

I wonder what the problem is. As far as I can tell, all 4 nameservers for the domain factorio.com are providing valid information, yet my local ISP's nameservers are not providing any information for anything inside the factorio.com domain.

Is my guess correct that this is a TTL (time to live) issue?

Do my local ISP's nameservers have an out of date NS record stored in their cache? I guess not, because querying them for the NS record also fails. Maybe it is an issue of negative caching?
Last edited by Tekky on Thu Nov 22, 2018 6:17 am, edited 4 times in total.

Jap2.0
Smart Inserter
Smart Inserter
Posts: 1748
Joined: Tue Jun 20, 2017 12:02 am

Re: Factorio Services Outage

Post by Jap2.0 » Thu Nov 22, 2018 5:11 am

How are you expecting people to see this if they can't get to *.factorio.com?

Otherwise, I can confirm that changing DNS works (and I've been meaning to do that for quite some time anyway), so thank you for the update on this.
There are 10 types of people: those who get this joke and those who don't.

User avatar
Nova
Filter Inserter
Filter Inserter
Posts: 861
Joined: Mon Mar 04, 2013 12:13 am

Re: Factorio Services Outage

Post by Nova » Thu Nov 22, 2018 5:35 am

I suggest to make a tweet about these problems in the future. Twitter would be the first place to search for informations about offline websites for me. ^^
Greetings, Nova.
Factorio is one of the greatest games I ever played, with one of the best developers I ever heard of. Image

Tekky
Filter Inserter
Filter Inserter
Posts: 859
Joined: Sun Jul 31, 2016 10:53 am

Re: Factorio Services Outage

Post by Tekky » Thu Nov 22, 2018 5:48 am

Jap2.0 wrote:
Thu Nov 22, 2018 5:11 am
How are you expecting people to see this if they can't get to *.factorio.com?
By manually querying the authoritative nameservers and adding the returned IP addresses to the hosts file?

Well, it worked with me. :)

User avatar
HanziQ
Factorio Staff
Factorio Staff
Posts: 630
Joined: Fri Mar 27, 2015 7:07 am

Re: Factorio Services Outage

Post by HanziQ » Thu Nov 22, 2018 9:41 am

Jap2.0 wrote:
Thu Nov 22, 2018 5:11 am
How are you expecting people to see this if they can't get to *.factorio.com?
Judging by the traffic on our servers and Heroku apps, not many people were affected, or for example, they might be able to visit the forums, but not the mod portal.
Tekky wrote:
Thu Nov 22, 2018 5:05 am
I wonder what the problem is. As far as I can tell, all 4 nameservers for the domain factorio.com are providing valid information, yet my local ISP's nameservers are not providing any information for anything inside the factorio.com domain.

Is my guess correct that this is a TTL (time to live) issue?

Do my local ISP's nameservers have an out of date NS record stored in their cache? I guess not, because querying them for the NS record also fails. Maybe it is an issue of negative caching?
It's an issue with DNSSEC and its very long (24h) TTLs.

BenSeidel
Filter Inserter
Filter Inserter
Posts: 513
Joined: Tue Jun 28, 2016 1:44 am

Re: Factorio Services Outage

Post by BenSeidel » Thu Nov 22, 2018 9:53 am

HanziQ wrote:
Thu Nov 22, 2018 9:41 am
It's an issue with DNSSEC and its very long (24h) TTLs.
Does anyone else remember when 48 or 72 hours were the norm? I mean, 24 hours was considered "average" at one time. I have had to keep old mail servers up for a week just to ensure that bad DNS records are flushed during a server move. Are you saying that it's an issue with their standard practices?

User avatar
HanziQ
Factorio Staff
Factorio Staff
Posts: 630
Joined: Fri Mar 27, 2015 7:07 am

Re: Factorio Services Outage

Post by HanziQ » Thu Nov 22, 2018 9:59 am

BenSeidel wrote:
Thu Nov 22, 2018 9:53 am
HanziQ wrote:
Thu Nov 22, 2018 9:41 am
It's an issue with DNSSEC and its very long (24h) TTLs.
Does anyone else remember when 48 or 72 hours were the norm? I mean, 24 hours was considered "average" at one time. I have had to keep old mail servers up for a week just to ensure that bad DNS records are flushed during a server move. Are you saying that it's an issue with their standard practices?
I'm not sure, I would've expected disabling DNSSEC where your DNS host is also the domain registrar to go smoothly, so the registry record is removed, so everyone stops checking the signature and remove the DS records like a day later, or something.

BenSeidel
Filter Inserter
Filter Inserter
Posts: 513
Joined: Tue Jun 28, 2016 1:44 am

Re: Factorio Services Outage

Post by BenSeidel » Thu Nov 22, 2018 10:14 am

HanziQ wrote:
Thu Nov 22, 2018 9:59 am
I'm not sure, I would've expected disabling DNSSEC where your DNS host is also the domain registrar to go smoothly, so the registry record is removed, so everyone stops checking the signature and remove the DS records like a day later, or something.
Agreed, but always assume that someone out there is using the worst-case settings, the ones that make you do the most amount of work.
Monitor your services and only remove your sever/routing off the old IP once traffic has stopped. Otherwise you end up in this situation.

I'm talking from experience... I've been chewed out for other peoples bad settings, even going so far as having to tell the other party how to clear their DNS cache. Go Team!

User avatar
HanziQ
Factorio Staff
Factorio Staff
Posts: 630
Joined: Fri Mar 27, 2015 7:07 am

Re: Factorio Services Outage

Post by HanziQ » Thu Nov 22, 2018 10:25 am

The problem is, that everything is still available on the old versions of any records that changed. The forums A record didn't even change, the whole thing is just some conflict while disabling DNSSEC.

BenSeidel
Filter Inserter
Filter Inserter
Posts: 513
Joined: Tue Jun 28, 2016 1:44 am

Re: Factorio Services Outage

Post by BenSeidel » Thu Nov 22, 2018 10:42 am

So I take it you disabled both the DNSSEC at the registrar level and at the nameserver level at the same time then?

User avatar
HanziQ
Factorio Staff
Factorio Staff
Posts: 630
Joined: Fri Mar 27, 2015 7:07 am

Re: Factorio Services Outage

Post by HanziQ » Thu Nov 22, 2018 10:50 am

BenSeidel wrote:
Thu Nov 22, 2018 10:42 am
So I take it you disabled both the DNSSEC at the registrar level and at the nameserver level at the same time then?
Yes, that was the only option available to me from DNSimple, a single button.

BenSeidel
Filter Inserter
Filter Inserter
Posts: 513
Joined: Tue Jun 28, 2016 1:44 am

Re: Factorio Services Outage

Post by BenSeidel » Thu Nov 22, 2018 10:52 am

I sympathise with you.

featherwinglove
Filter Inserter
Filter Inserter
Posts: 396
Joined: Sat Jun 25, 2016 6:14 am

Re: Factorio Services Outage

Post by featherwinglove » Fri Nov 23, 2018 8:16 am

Nova wrote:
Thu Nov 22, 2018 5:35 am
I suggest to make a tweet about these problems in the future. Twitter would be the first place to search for informations about offline websites for me. ^^
Not for me. I have been IP banned from Twitter and haven't been able to use it properly in over three years. This predates the conservative purge of recent months so I guess they really hate me. (OTOH, my Twitter feed hasn't been deleted, so you can go see what happened.)

Post Reply

Return to “News”