Wiki dump

Anything related to the content on our wiki (https://wiki.factorio.com/)

Moderator: Bilka

wanne
Inserter
Inserter
Posts: 45
Joined: Tue Jan 28, 2020 7:24 am
Contact:

Wiki dump

Post by wanne »

I know that it isn't that easy to do dumps with media wiki. So I ask: Am I allowed to do a dump it with httrack myself? Could do it gently with a few hundred kbit/s. Should I provide dumps so that other crawlers do not need to crawl over it again?
eugenekay
Smart Inserter
Smart Inserter
Posts: 1065
Joined: Tue May 15, 2018 2:14 am
Contact:

Re: Wiki dump

Post by eugenekay »

Special:Export appears to be enabled. The Wiki uses CloudFlare so try not to exceed any Rate Limits.

Good Luck!
wanne
Inserter
Inserter
Posts: 45
Joined: Tue Jan 28, 2020 7:24 am
Contact:

Re: Wiki dump

Post by wanne »

I rather extract the MD from html than dealing with the XML.

CloudFlare is there to keep out unwanted humans. Crawlers that do not execute javasrtipt, do not keep cookies and can change their IP do not tend to have that much of a problem.
But it was kind of the reason why I was asking. If it would be a stupid nginx I would not have feared to break anything.
User avatar
Sanqui
Factorio Staff
Factorio Staff
Posts: 398
Joined: Mon May 07, 2018 7:22 pm
Contact:

Re: Wiki dump

Post by Sanqui »

This is acceptable, just don't overload the server and follow the license (CC BY-NC-SA). :)
ovo
Post Reply

Return to “Wiki Talk”