Missing API Data

[noderunner]

I'm trying to get a list of all the mods and their licenses, however I'm having problems with the API.

First I get a full list of mods using:

Code: Select all

mods="$(curl https://mods.factorio.com/api/mods?page_size=max | jq --raw-output '.results[] | .name')"

Then I iterate through this list, getting the full details for all 8000+ mods. (This takes awhile!)

Code: Select all

printf $mods | while read i; do curl https://mods.factorio.com/api/mods/"$i"/full | jq '.' > /home/jdf/mods/$i.json; done

Now I have individual json files, I want to combine them.

Code: Select all

jq --slurp '.' /home/jdf/mods/*.json > /home/jdf/mods/concatenate.json

Everything is great at this point, we have full details for every mod on the Mod Portal.

Code: Select all

jq --compact-output '.[] | {(.name): .license.name}' /home/jdf/mods/concatenate.json > /home/jdf/mods/concatenate-licenses.json

Now we have a list of every internal name + license, but there's a problem, a lot of these mods are showing they have no license, for example:

Code: Select all

{"VehicleGrids-Reduced":"GNU LGPLv3"}
{"vehiclehud":null}
{"vehicle_physics_again":null}
{"vehicle-physics":null}
{"vehicle_physics":null}
{"Vehicle_Radar":"MIT"}
{"vehicles-equipement-grid":"MIT"}
{"VehiclesEquipement":"MIT"}
{"VehicleSnap":"MIT"}
{"VehicleTrails":null}
{"Vehicle-turrets":"MIT"}
{"VehicleWagon2":null}
{"Vehicle Wagon":"MIT"}
{"Vehicle_Zoom":null}
{"verbosity":"MIT"}
{"VersepellesAlienFarm":"Attribution-NonCommercial 4.0 International"}

When I check the mod portal for these "null" license mods, for example Vehicle_Zoom, I can clearly see that it does have a license (MIT).

Why then is the API missing these licenses? Why is there no easy way to sort or filter mods by attributes like their license on the Mod Portal? Why do I have to send 8000+ requests just to get incomplete data? I don't know the answer to these questions, but it keeps me up at night.

[vinzenz]

thanks for the detailed report!

Why then is the API missing these licenses?

This is a bug, which I'll fix soon(ish)

Why is there no easy way to sort or filter mods by attributes like their license on the Mod Portal?

Feature request noted

Sorry for keeping you up at night.

[vinzenz]

Why then is the API missing these licenses? Why is there no easy way to sort or filter mods by attributes like their license on the Mod Portal? Why do I have to send 8000+ requests just to get incomplete data? I don't know the answer to these questions, but it keeps me up at night.

The mod portal API now reports the correct license information