[MOD 1.1] Xander Mod v3.6.1

[Repofme1]

Hi all -

Phew. All my classes for the winter are finally over, and I'm a couple days into my job for the spring and summer. Still getting the hang of things of course but feeling good. Anyway, now that I don't have to deal with homework, I should finally have more time for the mod! Mainly on weekends but also a bit in the evenings. Some foundation-level things on my mind lately:

I'm seriously considering the possibility of splitting the mod into a core mod and multiple part mods. Mainly for the sake of data, as it is now I can't make much progress because half of the features I want need more graphics, and I only have a few megabytes spare in the current version before the mod portal doesn't like it. Not a problem for recipes and such, but images, and I don't really see enough functions to construct all my sprites from base assets via the code. Masking and color tinting can do some things, of course, but I can't do anything with, for example, brightness and contrast. I can't convert color to grayscale, I can't add noise, there are a whole mess of very useful functions that I have in Paint.NET but not the Lua code. If (and it's looking more like when) I convert to multiple parts, the current "xander-mod_X.Y.Z" will be used as the core or header, and will require all the individual parts. By themselves, these parts may end up adding stuff, doing nothing, or even failing to load, because I probably won't have time to write all the alternate recipes, items, substitutions, functions, etc. needed to make them capable of standing alone.

With the new resource set I mentioned a while back, worlds will likely be broken. However since there are the same number after as before, I technically could try to write a migration. I'm wondering if this is worth the effort, though, since all the mines and processing chains will have to be rebuilt anyway due to the changes in ore composition. Evaporites will have an especially bad problem,being converted from mineral water, so there will only be single tiles of it around where mineral water wells used to be. Heck, migrating a fluid to a solid might not even work at all, I don't know. Due to all this I think I'll not migrate resources, and redirect the effort to somewhere more useful.

Lastly, I have to say that I'm impressed beyond all expectation by everyone's determination! This community has shown some incredible grit in hammering through my crazy mod, I mean, 130 MW in coal -> steam, dang! And even without the basic turbine, too, because I messed up the parts and identity! It really encourages me : )

[eradicator]

Hey, good too hear you're back safe and sound :).

I wasn't aware that the mod portal has some sort of size limitation on files (not done anything bigger than a few megs) but that shouldn't be a too large problem (also now i finally understand why the big mods are all broken up into parts - something i don't like at all, mainly because factorio has no proper user-features to auto-download dependencies and many of the big mods declare their dependencies wrong).

The first thing that comes to mind is ofc using tinted masks for generic things like assemblers. But you'd have to use a single mask for all assemblers for it to be a space benefit, and you said you need some things that can't be done with masks at all so...scrap that.

Second would be to use image quantization tools (like this) to reduce the overall size of total textures which hopefully shouldn't be visible.

Third is like you mentioned splitting the textures into a seperate mod. Factorio makes this quite easy btw (i suggested this already when i first came to this thread :P). You could still keep all the code in one mod and split off only the textures into seperate mods. That way the texture mods would only need to contain info.json and the textures, no need for data/control/whatever.lua. This works because factorio always refers to textures by an abstract path like "__xander-mod__/textures" which you can just replace by "__xander-mod-texture-pack-1__/textures".

Also i've had some nice ideas on how to prevent having wrong recipe/assembler crafting categories on a code level to prevent "user"(==modder) error, hopefully i'll have some time to try them.

For the migration issue i'd still say: just do it. For 15 resources that's 15 lines of code which hurt nobody. Production lines are going to break either way, but at least people still get the chance to rebuild without starting from scratch. And don't end up with a map completely devoid of resources.

(Are you on irc btw? It's a good place for some quick answers :P)

[Therax]

Second would be to use image quantization tools (like this) to reduce the overall size of total textures which hopefully shouldn't be visible.

I ran 1.5.1 through pinga and immediately reduced the size from 149 MB to 119 MB, so PNG optimization is definitely a worthwhile step, whether textures are split into a separate mod or not.

If you're going to split textures out anyway, can I recommend that you religiously adhere to the file path naming conventions used by the base mod, at least for those entities that are simple recolors of base mod entities? That would make it simpler to auto generate entity records by a simple search-and-replace of entity names.

[eradicator]

Second would be to use image quantization tools (like this) to reduce the overall size of total textures which hopefully shouldn't be visible.

I ran 1.5.1 through pinga and immediately reduced the size from 149 MB to 119 MB, so PNG optimization is definitely a worthwhile step, whether textures are split into a separate mod or not.

20% reduction? That looks odd. I ran pngopt over a few of the files and got 10% on average. Lossless that is. Which means yours is lossy, but for lossy 20% is pretty low. Well, i guess it depends on the texture but i'd expect 50% from a proper quantization.

If you're going to split textures out anyway, can I recommend that you religiously adhere to the file path naming conventions used by the base mod, at least for those entities that are simple recolors of base mod entities? That would make it simpler to auto generate entity records by a simple search-and-replace of entity names.

Not sure what the path scheme has to do with that? Albeit he already seems to mimick the base pathes. But the correct approach would be to simply define the path prefix as a global variable.

XANDER_ASSEMBLER_SPRITE_PATH_ROOT = '__xander-mod-texture-pack-01__/'
XANDER_DRILL_SPRITE_PATH_ROOT = '__xander-mod-texture-pack-02__/'

Then use only those in the actual prototype definitions.

drill.pictures.filename = XANDER_DRILL_SPRITE_PATH_ROOT .. 'entiy/mining-drill-n/west.png'

That means if you ever need to move the textures somewhere else you just have to change a single global variable. Unless the unlikely case happens where you need two extra mods just for drill pictures :P

That discussion seems more fit to github though :P

[Therax]

20% reduction? That looks odd. I ran pngopt over a few of the files and got 10% on average. Lossless that is. Which means yours is lossy, but for lossy 20% is pretty low. Well, i guess it depends on the texture but i'd expect 50% from a proper quantization.

Some of the PNGs are already optimized, and I got 0% savings. For others there were 50%+ savings. Over all the files it was a 20% savings.

Not sure what the path scheme has to do with that? Albeit he already seems to mimick the base pathes. But the correct approach would be to simply define the path prefix as a global variable.

They mimic, but they're not quite the same, which means everything has to be special cased. For example, the vanilla assembling machine has path:

Code: Select all

graphics/entity/assembling-machine-1/assembling-machine-1.png

And the XM AM4 has the path:

Code: Select all

graphics/entity/production/assembling-machine-4/4.png

So it's close, but it's not good enough to be able to automate with a search-and-replace of "assembling-machine-1" with "assembling-machine-4," for example.

[eradicator]

So it's close, but it's not good enough to be able to automate with a search-and-replace of "assembling-machine-1" with "assembling-machine-4," for example.

What exactly are you even trying to automate? *confused*

[Repofme1]

Hi there -

Thanks for the replies and ideas, because (again) it's helped me see some flaws and improve my plans. I was thinking of splitting code and everything, but really that doesn't make any sense; the graphics take up 99% of the data in the mod! I remember you suggesting it, eradicator, but I can't recall why I decided not to do it then, and now we need it. Oh well, live and learn. Since the XM graphics already take up a full 150+ MB, I might need two graphics pack mods (to handle new things). And if I'm going to use more than one I might as well have several, so it seems like a 10 or 20% reduction in space by compression won't be super worthwhile. Especially considering the naturally low resolution of the game, where otherwise subtle changes to pixels are more noticeable. At first it seems simple to just make several of generic "xm-graphics-pack-n" tohold everything, but going through the data again, I realized that the entity graphics take up 85% of the graphics budget. So here's what I think: keep the item, fluid icon, tile, category, and other misc graphics in the core mod, and make additional packs for entity graphics. The misc stuff besides entities is only ~20 MB, which I think leaves plenty of room for expanding new tiles, items, recipes, etc. Then for the entities I can have generic "n" packs to take care of their expansion. I do really like how Factorio lets you specify an abstract root path, because as mentioned, it makes relocating graphics quite easy.
(I do like how Factorio lets you specify a root path)

A word on file organization: I set up my files with different folder structures because my mind has a lot of trouble navigating the base game's lopsided structure. I've found things make the most sense to me with about 5-20 items (be they files or subfolders) in each folder; if there are too few, then the file hierarchy gets way too deep to navigate, but too many, and you have to scroll for ages through each folder and squint to find what you're looking for. The base entities, icons, and technologies are especially difficult, I have to search everything I want instead of being able to rapidly click through. I guess this is bad for code and file automation, unfortunately, but I don't really know of a good compromise.

And yeah, a resource migration file will be very simple; it's good to confirm that it's a good idea as my gut instinct is telling me.

I'm not on IRC; I do have a Discord account that I used once (when Ironchefpython helped me set up GitHub). What I should do is just open it up whenever I'm working on the mod, but I always forget that it exists

[eradicator]

I remember you suggesting it, eradicator, but I can't recall why I decided not to do it then, and now we need it.

I do :P. I initially suggested it to reduce the needed upload filesize for small fixes. And you said that you often change icon so you'd have to always reupload the graphics anyway, so i didn't pursue it further. Ofc now you've reached the right conclusion that you can just leave the small things that often change (icon) in the main mod and put the large textures that don't change often into seperate packs that (almost?) never need to be updated.

Btw, personally i don't play xander for the graphics, so i'd be kinda interested in a comparison picture to understand why you said that tinted masks are insufficienct for you graphical needs. As i still feel that just switching to masks would solve most of those problems and also probably save large amounts of vram.

As for the data-structure: Personally in my mods i use a completely different structure from the base game. Because ultimately i'm the main user of the structure, so i see no reason to not use something that i feel most comfortable with. I.e. you're not the only one who finds the base-game structure uncomfortable :P.

[Therax]

What exactly are you even trying to automate? *confused*

Construction of Lua records in the prototypes folder, pulled in via data.lua.

I think it's easier to read and maintain something like:

Code: Select all

copy{
  from = "assembling-machine-2",
  to = "assembling-machine-4",
  overrides = {
    crafting_speed = 5,
    energy_usage = "400kW",    
  },
}

rather than have a big Lua table where almost every field is a copy-paste from some entity in base. Opinions may vary. If you're using a system like that, the more similar the modded entity is to the base entity, the more concise and readable the code becomes. For a mod that defines totally new entities, I don't think it matters, but a large number of XM's entities are stat-tweaked base entities.

I contributed such a system as xmutil.clone(), and I hope it's been helpful to Repofme1. If the graphics file structure followed the base game file structure more closely, however, many of the entity definitions in the Lua could be cut in size by ~50%.

too many, and you have to scroll for ages through each folder and squint to find what you're looking for. The base entities, icons, and technologies are especially difficult, I have to search everything I want instead of being able to rapidly click through.

I find it quite regular. Entities are under "graphics/entity/<entity-name>". Technology icons are "graphics/technology/<technology-name>.png". Item icons are "graphics/icons/<item-name>.png". The other icons are I admit a bit of a mess. I never scroll through folders at all. If I'm looking for a particular entity graphic, I open the entity folder and start typing the name, and hit Enter, and there I am.

I actually have a much harder time with XM's multi-level categorization. Why are etched but unassembled circuit boards under "board", but unetched boards are under "electric-1"? Why are insulators under "electric-1", but transformers are under "electric-2"?

Let's say I'm looking for the icon for "silicon-boule". Where is it?

I use it to build circuits, so maybe it's in item/part/electric-1 (or -2, or -3)? Nope.
Maybe it's under item/part/data, next to the sliced wafers? Nope.
Maybe it's not a part at all, but a component of the materials tree, so let's check item/material/nonmetal? Nope.

Eventually I had to search the .lua files in my editor to find out it's actually classified as an alloy, despite being nonmetallic. I would have been much faster opening graphics/icons and then typing "sil" to jump straight to it. Again, personal workflows and what seems "natural" varies by the individual.

As for the data-structure: Personally in my mods i use a completely different structure from the base game. Because ultimately i'm the main user of the structure, so i see no reason to not use something that i feel most comfortable with. I.e. you're not the only one who finds the base-game structure uncomfortable .

When I'm modding I spend most of my time writing and then debugging the .lua files, so I find it's worth a bit of pain to have an awkward file structure if that makes the scripting (i.e. data tables) simpler to read and write. I would think that goes double in something that modifies as many base game objects and defines as many new game objects as XM does.

[eradicator]

I totally agree on the copy() function. Infact i have exactly that with some additional funcionality in my main mod, which allows me to specify a simply copy of an entity, including new item and recipe, in about 8ish lines of code (excluding modification of the new entity). For graphics in xander i'd probably use a completely seperate file though i think. Graphics tend to have weird quirks with what you need to specify for what (i.e. picture or pictures or animation or animations etc) so i'm not sure how feasible auto-generating the paths for that is. And it would only ever work if the sprite sheets have the same pixel size as the base ones, otherwise you need to speficy those again and are back in the mess. So. Have prototype properties in one file, and everything that specifies graphic files paths in another would be my preferred way. Because it makes the actual prototypes much easier to read.

I haven't looked at xanders subfolder structure that much so i didn't realize it was that fragmented. @silicon-boule.

As for the data-structure: Personally in my mods i use a completely different structure from the base game. Because ultimately i'm the main user of the structure, so i see no reason to not use something that i feel most comfortable with. I.e. you're not the only one who finds the base-game structure uncomfortable :P.

When I'm modding I spend most of my time writing and then debugging the .lua files, so I find it's worth a bit of pain to have an awkward file structure if that makes the scripting (i.e. data tables) simpler to read and write. I would think that goes double in something that modifies as many base game objects and defines as many new game objects as XM does. :D

What if you can have non-awkward and still easy to script :P. My mods don't have nearly as many entities as xander has so my approach might not be applicable, but copying the base strucute isn't the only way to have easy-to-script things :D.

[Therax]

And it would only ever work if the sprite sheets have the same pixel size as the base ones, otherwise you need to speficy those again and are back in the mess. So. Have prototype properties in one file, and everything that specifies graphic files paths in another would be my preferred way.

Agreed. I only make the suggestion because as of right now so many of XM's buildings are simple recolors of base game buildings, between the inserters, machining assemblers, ore washing plants, etc. Once we start having different sprite sizes or something like your suggestion of tinted mask layers, then your system of separate graphics definitions is much more flexible and makes a lot of sense.

[riqotta]

Btw, Diamond Crystal has diamond boule as an ingredient twice. And inserters will only insert diamond boules for the first one, not the second. So it only works if you insert them manually.

Still going strong here . Thanks for the tip about the turbines being swapped around, I have all the ingredients for the more advanced recipe now, but I didn't earlier. I swapped my steam power to superheating boilers and medium temp steam turbines, now it outputs 220MW, and my solar is up to 320MW. I'm 160 hours in (I have biters off for this play through btw). Halfway through Assembly Machine 4 research at the moment. Creating production science packs at a reasonable rate is a challenge but my factory is slowly getting better at it.

[riqotta]

Finally researched fuel-grade uranium enrichment. Unfortunately, as I expected, two of the recipes unlocked can't be made. 238-Uranium hexaflouride hydrolysis and Fuel-Grade Uranium hexaflouride hydrolysis both have 2 fluid inputs and outputs but want to be made in an electric refining furnace which doesn't support that.

Anyone have thoughts on the fix? I might edit it to be in the chemistry category for now.

[abbayarra]

I just wanted to say thank you. I am a chemist and love that angle on this mod. I am loving playing you mod.

[Repofme1]

Hi all -

So, once again it took me over a month to get something really workable out, but at least I have a good feeling about it.

Summary:
New resource set, game progression temporarily broken because I haven't fixed any of the recipes yet
I'm working more efficiently with lists and loops to build the mod
Some graphics moved to a new mod, Xander Mod Graphics Pack 0, which is a required dependency for XM 2.0.0
NOTICE Bug found (already): fail to load with high-res graphics because of a few underscores where there should be dashes. Temporary workaround: if the game says "failed to load because (file with xander_mod) not found", just choose the button to disable selected mods, then change the sprite resolution to normal instead of high.

With 8 months now having past since I first published the mod, a lot has happened! Your enthusiasm for the mod greatly exceeded my expectations, and keeping up with it all quickly got out of hand for me. Other things that didn't help were class and life taking up much my time, low motivation due to mostly working on the mod and not having time to play & test it, and my own limitations when it came to efficiently coding and effectively working with other people. I honestly wish I had collaborated a lot more, and I feel like I've let you down, especially the people who had so many ideas and offered to help out. I guess I'm just too particular about how I want to make the gameplay, or organize my files, or most things. That, and my slow communication abilities make it difficult to coordinate changes well

So anyway, I've been itching to take a step back and revise the mod, which is what I'm trying to start now. Not completely starting from scratch but overhauling most of it. A month ago I went to start a world, but then I remembered I had thought of a whole bunch of changes to the resource system, that I needed to do before starting a new world, and well... That's why the 2.0.0 update is not really a functional mod. It hs the new resources but the old everything else. But the foundation is solid to build on, and my mind must have had some kind of hidden breakthrough because suddenly I'm able to code the mod with lists and for loops and stuff instead of doing literally everything manually. I know people were pointing me to this, but I must not have been ready, because one day it just happened. Another improvement that is now automatically working itself into my mod development is recording all of the parameters and sources for everything, like first principles, so I can reconstruct everything in a repeatable way when I want to modify it.

Some more details about the changes: I did write a basic migration to convert the old ones into new, but I recommend against continuing XM 1.5.1 worlds with 2.0.0. Mainly because fluid mineral water is now solid evaporites, so there will be hardly any tiles of it, but also because the new resource items don't work in the old recipes, and a bunch of the products or elements have been shuffled around. For reference, here's a list of the conversions (I think I posted a version of this earlier but it probably changed):
XM 1 New ID New Name Products
(Coal) coal Coal Seam Coal (carbon, hydrogen, sulfur), possible rare earths or uranium
(Copper Ore) copper-ore Porphry Copper Deposit Copper, sulfur, cobalt, gold
(Crude Oil) crude-oil Crude Oil Crude oil (carbon, hydrogen, sulfur)
(Mineral Water) evaporites Evaporite Salt Formation Salt, saltpeter, borax, lithium chloride, sulfur
(Sulfidic Ore) igneous-sulfide Igneous Sulfide Ore Iron, copper, nickel, sulfur, platinum
(Iron Ore) iron-ore Banded Iron Formation Iron, manganese
(Bauxite) laterite Laterite Deposit Clay, bauxite, nickel, maybe gold
(Garnierite) magnetic Magnetic Ore Iron, chromium, vanadium, platinum
(Lead Ore) massive-sulfide Mass Sulfide Ore Zinc, lead, silver, sulfur
(Natural Gas) natural-gas Natural Gas Natural gas (carbon, hydrogen, sulfur)
(Apatite) phosphorite Phosphorite Deposit Phosphorus, fluorine, calcium
(Heavy Mineral Sand) sand Economic Sand Deposit Silica sand,iron, titanium, zirconium
(Granitic Ore) skarn Skarn-Type Ore Tin, tungsten, maybe molybdenum, maybe gold
(Stone) stone Economic Stone Formation Limestone, quartzite, graphite, maybe magnesium
(Pitchblende) uranium-ore Pegmatite-Type Ore Uranium, rare earths, phosphorus, tantalum, maybe zirconium and lithium

Water N/A Surface Water Hydrogen, oxygen, salt, magnesium
Air N/A Compressed Air Nitrogen, oxygen, carbon dioxide, pure water

Xander Mod 2 also begins the long-awaited project of offloading the bulk of the graphics into a separate graphics pack mod, so that I shouldn't have to worry about file size every time I want to update the mod. The new mod is called Xander Mod Graphics Pack 0, and is also available on the mod portal. The primary mod correspondingly requires this pack as a dependency, to avoid missing graphics errors.

[Repofme1]

I just wanted to say thank you. I am a chemist and love that angle on this mod. I am loving playing you mod.

Thank you! I'm glad to hear you like the chemistry in the mod It's one of my favorite subjects, especially the heavy-duty industrial type. I hope it keeps giving you enjoyment, and that my slow pace isn't too much of a problem

[Deadlock989]

NOTICE Bug found (already): fail to load with high-res graphics because of a few underscores where there should be dashes. Temporary workaround: if the game says "failed to load because (file with xander_mod) not found", just choose the button to disable selected mods, then change the sprite resolution to normal instead of high.

You don't get that option when a sprite resource is missing/mis-named. You just get a standard Windows error dialog and loading is then halted. I now have to manually edit mod-list.json to get my game to load at all, even to change resolution ...

In my overhaul mod (WIP, except I haven't touched it for a few weeks) I don't give high and normal resolution resources different names. Instead I put them in different folders (e.g. "graphics/high/entities" and "graphics/normal/entities" etc.), and then I set a global variable with the path to all resources. It means you set the path once then you never need to think about the filenames ever again, and for most normal-res sprites you can just copy and paste files between folders and resize them without having to rename them.

[jodokus31]

I'm looking forward to have a look into the mod again.
EDIT: Seems that the new resources should be added to RSO config

With 8 months now having past since I first published the mod, a lot has happened! Your enthusiasm for the mod greatly exceeded my expectations, and keeping up with it all quickly got out of hand for me. Other things that didn't help were class and life taking up much my time, low motivation due to mostly working on the mod and not having time to play & test it, and my own limitations when it came to efficiently coding and effectively working with other people. I honestly wish I had collaborated a lot more, and I feel like I've let you down, especially the people who had so many ideas and offered to help out. I guess I'm just too particular about how I want to make the gameplay, or organize my files, or most things. That, and my slow communication abilities make it difficult to coordinate changes well

Honestly, the coordination/collaborations and management is the hardest part of computer science. You can try to isolate some work/tasks for others, if it fits in your plan. But in the meantime, the alpha/betatesting (aka playing) is already a big thing, we can do

[eradicator]

New resource set, game progression temporarily broken because I haven't fixed any of the recipes yet

I don't usually openly disagree with mod authors, but this sounds like a very bad idea. Having a broken mod on the portal will alienate new players and old players with auto-update alike. I strongly suggest you think about depublishing it again until it at least basically works (judging from your wording alone, can't test...). Sadly the mod portal lacks a "beta/experimental" functionality that would make publishing things like this easier. @Deadlocks post about having to reset mod-settings just makes my feeling against this stronger. Maybe you could upload the "experimental" releases right here to the thread and only upload them to the mod portal once it's somewhat stable? Btw, have you thought about requesting a sub-forum for Xanders Mod? This thread now has 26 unsorted pages :P.

So, once again it took me over a month to get something really workable out, but at least I have a good feeling about it.

With 8 months now having past since I first published the mod, a lot has happened! Your enthusiasm for the mod greatly exceeded my expectations, and keeping up with it all quickly got out of hand for me. Other things that didn't help were class and life taking up much my time, low motivation due to mostly working on the mod and not having time to play & test it, and my own limitations when it came to efficiently coding and effectively working with other people. I honestly wish I had collaborated a lot more, and I feel like I've let you down, especially the people who had so many ideas and offered to help out. I guess I'm just too particular about how I want to make the gameplay, or organize my files, or most things. That, and my slow communication abilities make it difficult to coordinate changes well :(

I've talked to and adviced countless new and old modders over the course of my forum live. And it's not at all uncommon that i write a longish - hopefully helpful - post just to have the modder give up and never finish the project. So someone like you who keeps working is pretty much the opposite of a "let down" ;). Keep up the good work. Everyone has motivational lows (i got one myself currently. I've wanted to make a large PR for month now :s).

Also grats on getting the grasp on loops. Everything is better with loops ;).

@Deadlock:
It strike me as odd that you would get a non-factorio error for what basically is an error in the data stage though, have you posted a bug report?

[ColeNOXyd2nd]

i am a big fan of this (honestly insane) mod. but i also disagree with how you managed this, just checked for updated mods, installed the update and was excited to see what changed... only to have my whole base gone. I would say if you are doing a complete rewrite you should maybe put it up as a second mod with a sub 1.0 version so people know its beta and there saves aren't effected.