Factorio Forums

Factorio Mods

Hello everyone! Last year I started working on a web app to host Factorio mods. I neglected it a little during the summer, and stopped working on it after college started! Now it's winter break (south hemisphere!) and I decided to git it the final touches for a decent launch!

Source code

The source is hosted on Github the app is made on Ruby on Rails and hosted on the Heroku free tier.

I think that in combination with the Factorio Mod Manager, by narrowtux which is also open source, now we have a pretty good combination to install mods!

How did you get the mods to the database?

The mods were all scraped from the forum (it's still missing some big mods though), and categorized manually (I'll ask for help establishing good intuitive categories later), and I only added the versions for 0.11.x, because time. At the moment there are 188 categorized and searchable mods in the database. When you open a mod on the app, there is actually an additional frame with the forum post, this turned out to be really useful since it usually has detailed instructions and you can make comments.

Where is the hosting?

To host the mods files I used a combination of hotlinking if the mod was hosted in the forum, and putting a mirror for an Amazon S3 bucket, because I'm scared of what the bill might look like. Ideally, all the mods would be hosted on Github, and use the Github release system to create downloadable zip files.

Also, the pictures are hosted by Imgur, because they're awesome, right?

I want to coordinate with the Factorio Mod Manager to register a protocol like `factoriomods://` with an URL of the mod download, so the mod manager can just pick it up and install it.

How to submit mods

For now the submission is not totally public, since I still feel I need to strengthen the system. However, since most mods were scrapped from the forum. If your mod it's in there and you want to manage it, or delete it, or add a new one, just PM me and I'll associate your account with the mods and mark you as a dev so you can also add new mods.

I'm surprised that less than 10% of the mods were hosted on Github. Github is awesome, and allows you to see the source online, fork the code, report bugs, discuss features, etc. And specially useful, host releases, which is nice for those 50mb mods.

I'm 100% open to suggestions and you can fork the project or create issues.

Features upcoming/planned/thinked:

- Multiple authors associated with mods. Because I noticed some mods were made by multiple people, and for some reason I didn't think about it.
- Authors page with a list of the mods.
- Bookmarking mods, and single-click-downloading multiple mods in your bookmarks page.
- Dependencies. Some mods have dependencies on the `info.json`. I would like to automatically establish dependencies between mods based on that file.
- Modpacks. Similarly of how bookmarks would work, I would like to be able to create modpacks with a list of different mods with specific versions.
- Github integration. I would like for devs to be able to link the Github project with the mod, so after they release it on Github it automatically updates on the webapp. It could also pick up the README from Github and add a description.
- Allow mods to be switched from "use forum iframe/use hand crafted description/use both". With the Github integration, updating the description would only be a `push` away.
- Some devs use AdFly links to link to their downloads. Maybe I could add an option that pops up the first time the user tries to download a mod, asking them if they want to activate "devs support links" or something.

And now, if you allow me, *I'm gonna go try 0.12.*

Very usefull and very nice website. Good job making it. I will defiently use it in the future to download mods!
I think the forum can learn a lot from your site of how to show off mods.

Hu. Woa. And I was so busy with setting up the new mod-categories, that I didn't watch this board.

That is, what I "warned" the devs that will happen: Others will do this job. This looks pretty cool. And nice. Did you already talked with the devs about this? cause I think they might be very interested...

I need to take me time to look this a bit deeper. Are you interested in critics?
I'll will pitch this in the appropriate way.

PS: What I like most are the rotating wheels in the banner.

Holy Crap! I had the same idea but I mainly develop in PHP.

I'm surprised that less than 10% of the mods were hosted on Github. Github is awesome, and allows you to see the source online, fork the code, report bugs, discuss features, etc. And specially useful, host releases, which is nice for those 50mb mods.

QFT. Also, if their is a way to auto add the mod tag + version to the zipfile based on the information in info.json, that'd be great

How do you scrape mod images? I noticed some mods don't have an image, even though there is a pictures spoiler. (air filtering for example).

If everyone were to tag their post with a mod header version (the same way the http protocol versions their headers, e.g. HTTP/1.1), you could adjust your scraper according to that tag. That way, if new things are added to mod ehaders, the mod header version updates and you know how to parse that specific version. See: https://forums.factorio.com/forum/vie ... 517#p95517

Funny, I was just thinking about doing something like this just the other day. Guess there is always somebody faster
The temptation to jump on board is high with this one, though I gotta say Ruby is a massive turn off.

As for mods not being available on github I also, complained about that recently. I'm hoping that a structured website like yours will make people more aware of the necessity of proper licensing and version control.

Oh wow, I got really used to Reddit and I forgot that I don't get notifications here haha. Sorry the delay!

ssilk wrote:Hu. Woa. And I was so busy with setting up the new mod-categories, that I didn't watch this board.

Haha yes, I saw, you messed up the scraper that was looking at just one subforum. I had to create a multi-subforum scraper, but it was fun, and now it works with concurrency

ssilk wrote: That is, what I "warned" the devs that will happen: Others will do this job. This looks pretty cool. And nice. Did you already talked with the devs about this? cause I think they might be very interested...

If by the devs you mean the Factorio devs, then not yet, but it would be nice to know what are the official plans for mods hosting and management!

ssilk wrote: I need to take me time to look this a bit deeper. Are you interested in critics?
I'll will pitch this in the appropriate way.

I'm very interested in critics! I love making useful web apps, and I want it to be as good as possible! I created a Trello board with the development progress, it's just me though, but hey, it's fun to move cards around! It has the "ideas" box on the left, all the way to the "done" box on the right.

I'm on winter break until August 10, then it's college again, so development will definitively slow down

ssilk wrote:PS: What I like most are the rotating wheels in the banner.

Yay! I'm glad someone noticed! It's definitely the best feature.

Talguy wrote:How do you scrape mod images? I noticed some mods don't have an image, even though there is a pictures spoiler. (air filtering for example).

If everyone were to tag their post with a mod header version (the same way the http protocol versions their headers, e.g. HTTP/1.1), you could adjust your scraper according to that tag. That way, if new things are added to mod ehaders, the mod header version updates and you know how to parse that specific version. See: https://forums.factorio.com/forum/vie ... 517#p95517

The scraping mechanism is not fully automated, I mean, the way the mods are presented vary wildly from post to post, so I can't make any assumptions. With the new template that ssilk wrote it might be feasible to automatically get all the data from the forum though. The posts themselves are not scraped yet, I started writing a post scraper but it wasn't worth it, this will probably change. The images I just picked them myself, it's just one picture per mod. I tried many things with the mods images, but using a single picture worked best, since I decided to iframe the forum post.

Gandalf wrote:Funny, I was just thinking about doing something like this just the other day. Guess there is always somebody faster
The temptation to jump on board is high with this one, though I gotta say Ruby is a massive turn off.

As for mods not being available on github I also, complained about that recently. I'm hoping that a structured website like yours will make people more aware of the necessity of proper licensing and version control.

Aww, sorry you don't like Ruby! It's such a beautiful language, I personally love working with Ruby and with Rails

Regarding the licensing, I'll add a licensing attribute so you can set the licence of the mod. I actually had it in place but decided to remove it because I didn't want to have so much overhead on the mod edition form, which is already really long.

I would like to link the mods to Github, they have a great API to read all kinds of information. The releases can easily be extracted from their API. For example here are the releases for DyTech. And I could also automatically look at the README for the description data. And I could also read the licence automatically from there.

Zequez wrote:[...]

Talguy wrote:How do you scrape mod images? I noticed some mods don't have an image, even though there is a pictures spoiler. (air filtering for example).

If everyone were to tag their post with a mod header version (the same way the http protocol versions their headers, e.g. HTTP/1.1), you could adjust your scraper according to that tag. That way, if new things are added to mod ehaders, the mod header version updates and you know how to parse that specific version. See: https://forums.factorio.com/forum/vie ... 517#p95517

The scraping mechanism is not fully automated, I mean, the way the mods are presented vary wildly from post to post, so I can't make any assumptions. With the new template that ssilk wrote it might be feasible to automatically get all the data from the forum though. The posts themselves are not scraped yet, I started writing a post scraper but it wasn't worth it, this will probably change. The images I just picked them myself, it's just one picture per mod. I tried many things with the mods images, but using a single picture worked best, since I decided to iframe the forum post.[...]

I saw from the source code you're parsing topic titles right now With the new mod header template it's going to be a lot easier I guess.

As for images, why not make a simple media list, i.e. 4 columns, multiple rows, and then add a lightbox so you can click the images. You could name the entity ModImage, with props title, description(WYSIWIG) url, width, height, ???

Just some remark about your API: The name field is blank everywhere, and I'm guessing you disabled that intentionally. What is the progress on the API and what are your milestones for them right now? Also, the code is kind of readable even for a backend PHP developer(me), but docs would be nice offcourse

You said you're still in school so don't rush it offcourse, but please don't let this become something like Duke nukem Forever

Zequez wrote:If by the devs you mean the Factorio devs, then not yet, but it would be nice to know what are the official plans for mods hosting and management!

I already said a bit via PM about it. There is also this thread:
https://forums.factorio.com/forum/vie ... =55&t=9210
https://forums.factorio.com/forum/vie ... =6&t=12199

The official statement is: Yes, they want to have a mod-db included. They want to have them not in php. (<shrug> - php with some framework around is in my opinion a good enough language for that kind of work, but ruby is also ok, I think ), and the minimal solution looks nearly as yours, despite, that it holds also all mod-downloads. <wink> <wink>

The idea I developed with them in Prague was: copy/paste the link to the download of a mod into Factorio and Factorio will download the mod and install it after some minimal check. That was the reason, why I added the download-link into the mod-header. And all the other stuff I tried to bring into the header has also some backgrounds there: Make it possible to "synchronize" forum and possible mod-db candidate.

I explained also a bit here: https://forums.factorio.com/forum/vie ... =10#p93873

I'm very interested in critics! I love making useful web apps, and I want it to be as good as possible! I created a Trello board with the development progress, it's just me though, but hey, it's fun to move cards around! It has the "ideas" box on the left, all the way to the "done" box on the right.

I'll try to cut some time out of my rips and make a full analysis, as I would make it for a customer. I always like to help young people, that are willing to learn.

Talguy wrote: I saw from the source code you're parsing topic titles right now With the new mod header template it's going to be a lot easier I guess.

As for images, why not make a simple media list, i.e. 4 columns, multiple rows, and then add a lightbox so you can click the images. You could name the entity ModImage, with props title, description(WYSIWIG) url, width, height, ???

Just some remark about your API: The name field is blank everywhere, and I'm guessing you disabled that intentionally. What is the progress on the API and what are your milestones for them right now? Also, the code is kind of readable even for a backend PHP developer(me), but docs would be nice offcourse

You said you're still in school so don't rush it offcourse, but please don't let this become something like Duke nukem Forever

Ahh yes, you reminded me, I added the API yesterday and the mod name attribute (from info.json) is blank for every mod, I forgot to make a script to automatically set the name based on the file name (the files are hosted on Amazon S3, so I can't just extract the info.json after I uploaded them), so every name is blank so far

I'll get it done today after I finish the bookmarking feature.

Regarding the images, I swear I did a lot of things with them:

- At first I hosted the images myself with their own models, you can still see that the "mod_assets" table still exists in schema.rb.
- It even had support for videos links at some point, but then I disabled that in favor of using gifs.
- I created a pretty cool lightbox that would zoom in and follow the mouse (the code it's still on "javascripts/lib/image_modal.js.coffee").
- Then I realized hosting the gifs myself would have been really expensive, and that it would be much more efficient to use "video-gifs" that are actually gifs converted to HTML5 videos, and that doing that myself would be totally out of the scope of the project.
- So I decided I would host all the images on Gfycat and Imgur. So I made a media_links class, which was on /lib/media_links.
- Then I realized my lightbox solution wouldn't work with the embedded videos from Gfycat.
- Then I realized that Imgur already supported .gifv, so no need to additional support Gfycat links.
- Then I decided to just host everything on Imgur and made an Imgur class, which it's still there somewhere in /lib
- I actually started researching the Imgur API.
- Then I had the idea to iframe the posts on mods#show because I couldn't really ensure that the mod in the database was 100% up to date.
- After all this I realized that I was spending too much time handling the images instead of doing stuff that was really useful, so I just added a field to paste a single Imgur link and use that as a single picture. Besides, with the iframe there, adding multiple images wouldn't really be critical, as long as it had a thumbnail for mods#index it would be alright.
- I'll definitively implement better images handling at some point, but it's not a top priority for now. I spend WAY to much rewriting this stuff, next time I'm gonna spend a lot more time planning before writing.

Zequez wrote:[...] I spend WAY to much rewriting this stuff, next time I'm gonna spend a lot more time planning before writing.

Lol, sounds familiar... Again, awesome work so far!

Sooo..., in the far far far future, maybe also a wide screen responsive version (24 columns instead of 12, an then two mod columns )

Your PM made it sounds like the mod api would take much longer ^^ Seems like its not to far away from working

http://api.factoriomods.com now exists

See https://trello.com/c/gyKmOC00

This is really awesome! props for open sourcing it as well.

Dude, this site is amazing and the fact, that it is programmed using rails, makes me happy, as I'm a rails developer myself

I just wanted to say, that I now may handle my mods myself on your site.

Also I just noticed a little typo:
After editing a mod for the first time, the little alert message at the top spells:

Alert with typo wrote:This mod is currently being reviewed. The process it's only done once for new users.

but it should be

Alert without typo wrote:This mod is currently being reviewed. The process is only done once for new users.

ssilk wrote: The idea I developed with them in Prague was: copy/paste the link to the download of a mod into Factorio and Factorio will download the mod and install it after some minimal check.

One problem with this - currently Factorio loads mods before it shows the main menu and any new mods added to the folder afterwards are not included in available mods so you must restart the game.

One way to overcome this is with a launcher (much like the minecraft one or the one by @Mclama - that he stoped working on for some reason) that manages profiles and mods in those profiles, where after you add/enable all the mods you want in a profile (or remove/disable the ones you don't want - like for a tutorial video) you launch the game and play - what do you think ?

P.s. Don't get me wrong - I'm all for pre-testing a mod to make sure it will not break the game with bad code

Kovarex never implemented features by adding provisory and less useful solutions. So - when you ask me - download via link means: You can install/deinstall a mod while the game runs.

jockeril wrote:One way to overcome this is with a launcher (much like the minecraft one or the one by @Mclama - that he stoped working on for some reason) that manages profiles and mods in those profiles, where after you add/enable all the mods you want in a profile (or remove/disable the ones you don't want - like for a tutorial video) you launch the game and play - what do you think ?

I was gone from the factorio scene for 7 months, and I've been playing more than anything
I tried to get the mods to move my thread a long time ago, but to no avail.

I'd have been planning on remaking the launcher.

And also, my site's 1 year expired, which the launcher was using for some features, Which is why the launcher is no longer starting, But before that it was still working with v12.3
I just now tested that it wasn't launching after the site went down, so i will make a new update to get it to at least work again, But keep in mind about the optional dependencies in it that are bugged out.

OBAMA MCLAMA wrote:
And also, my site's 1 year expired, which the launcher was using for some features, Which is why the launcher is no longer starting, But before that it was still working with v12.3
I just now tested that it wasn't launching after the site went down, so i will make a new update to get it to at least work again, But keep in mind about the optional dependencies in it that are bugged out.

Now I understand why it was working one day and not the next... thought I made a mistake when I moves the launcher & .json file to a dedicated folder... for now I'm using the other FactorioModManager which seems even less friendly at least yours had an easier interface to use - will you add an option to run different installs (for using different versions - for instance) ?

jockeril wrote:

OBAMA MCLAMA wrote:
And also, my site's 1 year expired, which the launcher was using for some features, Which is why the launcher is no longer starting, But before that it was still working with v12.3
I just now tested that it wasn't launching after the site went down, so i will make a new update to get it to at least work again, But keep in mind about the optional dependencies in it that are bugged out.

Now I understand why it was working one day and not the next... thought I made a mistake when I moves the launcher & .json file to a dedicated folder... for now I'm using the other FactorioModManager which seems even less friendly at least yours had an easier interface to use - will you add an option to run different installs (for using different versions - for instance) ?

i released a v0.4.8, tested it on my current 12.4 mod setup and it appears to be working good and fine right now.

Different install directorys, Well not with the current launcher but with a remake version, I could possibly add something like that. Interface is something i wanted to be really simple and not too complicated. And i wanted it to be easy to quickly switch profiles. So how about the profile has its own install directory? That might work out better.

Just a suggestion, language pack support would be useful. Everyone can upload a language.cfg to any mod, and these files will be shown at the mod's page