Friday Facts #105 - The Grey Zone

[Paranoimia]

What counts in the end is running code. Not text.

I completely disagree.

If you rely on code as documentation, it makes it even more difficult to learn about. Sure, it's more "accurate", as you say, but there's a much higher learning curve to it as well. The point of documentation is to make it easier to understand what the code does and, in the case of an API, how it can be used. Making a developer go the extra step and learn the intricacies of the code adds a huge barrier in getting them to use it.

In this case, though, you're probably right. The only reliable source of information about the modding API is probably the source code, which is bad. You may be fine looking at the source code, but that alienates a lot of developers that could be making Factorio even better with mods.

There is a reason that salesmen exist. Every industry has them, and they are an important part of attracting people to your product. It's a lot more difficult to sell something without having a salesman to help people understand why they need it. You just need good salesmen.

[ssilk]

If you rely on code as documentation

I said "In the end". Maybe this is a bit mistranslation to English, but for me it means, that when I'm in doubt, when I try to find a bug, when I test, misuse, don't understand, think I understood and now it fails ... then I look into the code (if available) and not in the documentation, cause the docs lie.
Always. I never found docs (of bigger projects, for small projects it is possible), that explained everything correctly.

That doesn't mean, that I NEVER look into documentation. That's just the first step. But when you come to really old code, 15 or 20 years old (and 60% of all code is such old code, older than 5 or 10 years or even older), the worth of the docs is in most cases just a good laugh. My experience with such docs is: Read it, then delete it (or better: move it to a place, where it doesn't disturb too much).

So when you see Factorio as such a long-time-project (now with more than 3 years!) then a good documentation is fine, good code (well written, easy readable, good to understand, concepts are not hidden in dubious side-effects) is much, much, much, much, much more important.

In other words: Good code doesn't need much docs.

And - O. K. - good tools can create good docs out of good code. Once I've been in a project, where we wrote our own code-analyser to generate docs from hundreds of input/output-masks. It was one command, to generate the docs again.
But what I mean is, that I saw a programmer, who wrote hundreds of lines of inline-documentation. I asked him: Why do you write a textual description of what you programmed? He said, because it is complex code, and it makes complex things. I asked: Then make it less complex; when you are finished, you can run the doc-generator and don't need to write docs. He didn't understand and what we did then was to rewrite a class with 1000 lines of code (without comments) to 5 classes with less than 100 lines. The resulting auto-generated API-Docu was clear and simple.
Then he understood.

[Paranoimia]

I agree that good code is extremely important, but I believe that good docs are needed as well. Not every little thing needs to be documented, that's definitely more effort than it's worth, but the functions and objects exposed to the public need to be explained properly.

At this point, this is just a discussion about documentation styles, which is getting off-topic. It's clear we have different backgrounds and opinions about code, so there's not much more to discuss relevant to the FF. I guess at this point, I am just waiting to hear for more documentation news from the devs.

[Oxyd]

snip

That is actually good feedback there. The documentation tool is still in its early stages, so I can take what you said into account without giving you the infamous reply of “Well, it's done now and we're not rewriting the entire thing”.

It's also easy to get stuck in a particular idea of doing things a certain way and not realising alternative ways. The Wiki is split into pages so that each page corresponds to one of the files in our code. Currently, the documentation tool does the same thing. I agree that that doesn't necessarily make sense for a user who has never worked with our codebase – i.e. the vast majority of the modders out there.

So how would you split the documentation up? Certainly it can't all be just one page, that would get super-messy. On the other hand, you feel that the current way of splitting it is too much. As someone who's trying to get into the API, what sort of organisation do you think would help you?

A wiki is not a good place for documentation.

Yes, absolutely. I personally hate it when I try to use a library, search for the documentation and they just throw a wiki at me.

[Paranoimia]

So how would you split the documentation up? Certainly it can't all be just one page, that would get super-messy. On the other hand, you feel that the current way of splitting it is too much.

Many popular libraries use the single-page approach with very good effect. The single-page style can make documentation linear; like a guided tour through what is available in the library. I'd say this style is more friendly to developers who are just starting out.

The Backbone.js documentation does this style very well. Backbone.js is an application framework, so they start off with an explanation of the strengths and goals of the framework (Getting Started, Introduction). After that, it has an overview of the over-arching concepts (the rest of Getting Started). Then they get into the details for each of its modules.

I like this style because it eases me into the documentation. It shows me the big picture right away, giving the broad overview of what it has to offer. After that is when it actually gets into the details, providing me with documentation to read in the order the devs want me to read it. It makes for an easy read, and doesn't need me to look around to figure out what I should be looking at next. Very good for starting out. And if I'm looking for specific information, the side-bar lets me quickly find a section, or I can even do a search through all of the documentation at once. Ctrl-F is a beautiful tool in this style of documentation.

Another documentation style could be to split the documentation into pages based on modules (or however you split up your code). This lets your documentation be more focused, since you're only showing information related to one topic at a time. I'd say this style is better for larger APIs, since it makes it easier to organize information.

Mozilla does this style well for the Firefox Extensions' APIs. It's not an entry page for their documentation, so they can get straight to the point and let the user find the section that they need. Then each sub-page contains everything needed to work with that portion of the API. The entry page they do use for the API is the page that is more friendly to developers who are just starting out, but can easily be skipped by people who know what they are doing.

I like this style because it organizes information, and the API as a whole, much better. It more clearer (and concisely) tells me what the high-level parts of the API are, and doesn't show me the details until I go looking for them. It makes it easier to include types of documentation (ie. guides, tutorials) other than straight reference material, since they can simply be their own pages. This style is oriented more towards developers who know the basics already.

...intermission

As someone who's trying to get into the API, what sort of organisation do you think would help you?

Now.. in this context, I agree that the single-page style wouldn't work overly well; the content [currently on the wiki] is too diverse to be put on a single-page. Likewise, having a separate page for each item (specifically, prototypes..) doesn't work well either; many pages aren't substantial enough to warrant navigating away to a new page. Splitting it up into pages makes sense, you just need to find out what each page should contain.

Keep related information together, and when not possible, clearly link it to relevant information. For example, from the wiki..

• * Related: Why are Prototype/Container and Prototype/SmartContainer on different pages? They can be included together, since SmartContainer is just an extension of Container.
  * Relevant: Prototype/ItemGroup is very relevant to Lua/Group, but neither page has a link to the other.

Most of the wiki pages are either Prototype pages or Lua Object pages. Even just including all that together as a 'Prototypes' and an 'Objects' page (in that single-page style we talked about ) would cut down tremendously on the page loading you need to do now with wiki. Then it's just about finding a good way to organize that information on a page, which isn't too bad.

Other than that, the main issues with the documentation is moreso about the readability/formatting and accuracy of the content, which is a whole different can of worms (but it not being on a wiki is a great step for that too).

Lastly..

It's also easy to get stuck in a particular idea of doing things a certain way and not realising alternative ways.

Take some time; do some research. See what other people are doing. What do you like about the library documentation that you use? What do you not like? Make it something that you're truly proud of.

While writing this reply, I found this blog post about designing API docs. I agree with what it says, and would recommended it for a read.

[ssilk]

So how would you split the documentation up? Certainly it can't all be just one page, that would get super-messy. On the other hand, you feel that the current way of splitting it is too much. As someone who's trying to get into the API, what sort of organisation do you think would help you?

I think this is a common mistake, that many documentation-writers are doing: they think, they have to make ONE documentation. One, that fits all needs. One, that includes every information of each standpoint.

I think Paranoimia made some quite good examples how to do it right. I would see that docs (and some more) as some kind of reference. Because they use a trick to write such good docs: they don't think, "oh good, I have such many information about the code and everything, how can I bring it to the users?" (Or similar thoughts)

They think: "Who needs to use the docs? What does he look for?"

They have created profiles for all the types of users they know. They analyze, for what the users are searching for and they do much more to get a picture of what the users want to know, when they come.

The user, that just wants to get an overview, one, who is searching for some special information, users thinking about changing from another framework, etcetera. There are many profiles of such users.
And what they do then is to link the information to the sight of one profile.
I repeat: link the information.

I know, it's not the best comparison, but I have tried to do that with the Factorio wiki main page (and some levels deeper). You can see my try to do this kind of thinking on what is the user searching for. (The results are of course not really exciting yet , but compared with the afford I put into it I think it's ok. Of course I would do it now completely different, but I wanted to give others also a chance to make something with the wiki)

So I try to repeat: In practice it is not possible to make one documentation, "that fits all". If you try, you'll need to make compromises ...
Instead it's much better to have different "views" to your docs. A (new?) Factorio programmer, don't want to read an introduction on each page how Factorio works. He wants to read it once of course. But then he just wants to read pure API specs, generated out of the code. And that's also a candidate for eventually reading the related c++ code then, just with a click. A newbie, which doesn't even know Lua is excited about docs that links to a Lua-introduction portal, eventually video tutorials of how to program for Factorio. A modder, that has made a complete overhaul of Factorio has a quite different need for information, than a frustrated gamer, who wants to fix a simple bug in a mod, which is essential for his current game.

In the end there is no "here is my documentation" and you can print it out as a book. The truth is more, that a good documentation has no defined start and no end, because it depends on what links you follow as reader.

And I also want to mention here the technical aspect. mediawiki (which is used for the factorio wiki) is incredible mighty, when we speak about how to display and link information. What we currently see in the Factorio wiki is using about 5% of what's really possible. For example: The same information can be showed on different pages and is looking quite different, depending on the context. Or a category selection on the left side is a plugin, that needs to configured. There is really much, which can be done.

One of the first things I would do is to separate forum and wiki into own services. Currently all is inside of one Apache instance, which leads to extremely long URLs and makes the wiki dependend on the forum and vice versa. It makes sense to make different wikis for the different needs and link the information (which is also well supported by media wiki). Otherwise you increase the side-effects and come into namespace conflicts.

After that, the first task for me was to create me a list of user-profiles, based on facts... And then we see...

[ratchetfreak]

One thing to help the in-game docs is to include the parameters and return type of each function.

It would explode the normal output of the help() command so I suggest to add an optional parameter with the name of the function and let it output the expected sig and maybe a short description of what it does.

Another thing is to hide the functions that you can't call on LuaEntities. This includes things like "neighbours" that only works on poles, inventory commands that only work on entities with inventories,...

@SSilk One thing against your code as doc argument is that we don't have access to the code. The only documentation we got is what the devs release in the wiki and the output of the help() function. Both of those are substandard as documentation TBH.

[ssilk]

I need to clarify: "code" is also, when generated. What we all know is of course automatic generated docs as you describe above (input and output parameters). But if you know the context of code, you can generate much more automatic information out of the code. What I don't mean is generated docs, that takes the higher level of information out of comments in the code, that are (in most cases) written by the programmer. That kind of docs tend to lie.
Of course this docs are needed, sometimes there is no way around them, but in my opinion this kind of doc is over evaluated, cause see the former article of me: the programmer tends to write docs for the wrong people. He thinks he needs to make compromises. But he don't! For his co-workers he don't need to make big comments. Cause they can read the code. And for anybody else it makes no sense to write much, cause anybody else will see this method or whatever in the context of the whole other stuff. Or if, he wants to know it more exactly, and then you can also put the whole function into the doc (I saw that also one time, but I think that is a bad idea).

One more example:
Once I had code, that had many functions, but generally all where the same, just bit different. The programmer made a comment in the first and explained how everything works (everything!) and in all others he said: look into this function, there is everything. Well, we can discuss, if that's good or bad idea, but it worked in that moment and everybody was more or less satisfied... , but the thing is, that things change.
Half a year later the functions changed. For some functions we need now a new backend and so they where nothing else than a wrapper. Still the comment said: look there, this is how it works. That was at that time generally true, but the backend keeps slowly evolving and in the end the functionality changes. Nobody cared about the wrong comments, cause everybody knows, it's only used in a small front end, and there only those functions, which hasn't changed and still follow the docs. In the end this lead to a complete disaster, cause new programmers, that didn't look into the code thought it works as described. So he took one of the functions, that changed the backend.

[Paranoimia]

they think, they have to make ONE documentation.

Yes, it's important to note that there are many different types of documentation. What we're talking about here is mostly Reference Documentation (since that's the only type of documentation that can be auto-generated). This is the documentation that mostly helps developers who already know the basics, but still needs to be clear enough for new developers to understand.

Other types of documentation (guides, tutorials) can be done later on. And I'm sure the community could help with those ones

That kind of docs tend to lie.

Only if the programmer lies. It always comes back to needing the programmer to do his job correctly. If he changed a public-facing section of the code without updating the docs to reflect the change, he is not doing his job correctly.

The two examples you've used up to now have not been about documentation. They've been about bad programmers either over-complicating their code or being lazy with documenting their code.