Missing API Data
Posted: Mon Aug 02, 2021 6:28 pm
I'm trying to get a list of all the mods and their licenses, however I'm having problems with the API.
First I get a full list of mods using:
Then I iterate through this list, getting the full details for all 8000+ mods. (This takes awhile!)
Now I have individual json files, I want to combine them.
Everything is great at this point, we have full details for every mod on the Mod Portal.
Now we have a list of every internal name + license, but there's a problem, a lot of these mods are showing they have no license, for example:
When I check the mod portal for these "null" license mods, for example Vehicle_Zoom, I can clearly see that it does have a license (MIT).
Why then is the API missing these licenses? Why is there no easy way to sort or filter mods by attributes like their license on the Mod Portal? Why do I have to send 8000+ requests just to get incomplete data? I don't know the answer to these questions, but it keeps me up at night.
First I get a full list of mods using:
Code: Select all
mods="$(curl https://mods.factorio.com/api/mods?page_size=max | jq --raw-output '.results[] | .name')"Code: Select all
printf $mods | while read i; do curl https://mods.factorio.com/api/mods/"$i"/full | jq '.' > /home/jdf/mods/$i.json; doneCode: Select all
jq --slurp '.' /home/jdf/mods/*.json > /home/jdf/mods/concatenate.jsonCode: Select all
jq --compact-output '.[] | {(.name): .license.name}' /home/jdf/mods/concatenate.json > /home/jdf/mods/concatenate-licenses.jsonCode: Select all
{"VehicleGrids-Reduced":"GNU LGPLv3"}
{"vehiclehud":null}
{"vehicle_physics_again":null}
{"vehicle-physics":null}
{"vehicle_physics":null}
{"Vehicle_Radar":"MIT"}
{"vehicles-equipement-grid":"MIT"}
{"VehiclesEquipement":"MIT"}
{"VehicleSnap":"MIT"}
{"VehicleTrails":null}
{"Vehicle-turrets":"MIT"}
{"VehicleWagon2":null}
{"Vehicle Wagon":"MIT"}
{"Vehicle_Zoom":null}
{"verbosity":"MIT"}
{"VersepellesAlienFarm":"Attribution-NonCommercial 4.0 International"}Why then is the API missing these licenses? Why is there no easy way to sort or filter mods by attributes like their license on the Mod Portal? Why do I have to send 8000+ requests just to get incomplete data? I don't know the answer to these questions, but it keeps me up at night.
