Friday Facts #366 - The only way to go fast, is to go well!

Regular reports on Factorio development.
Manual Inserter
Manual Inserter
Posts: 2
Joined: Fri Aug 02, 2019 6:26 pm

Re: Friday Facts #366 - The only way to go fast, is to go well!

Post by careful-player »

Re Test dependencies and Integration Tests

My solution for this problem is to treat this as a chain of tests. That is, I put the outputs of each test into some form of constant (code constants, test data factories, or files) and then read those as inputs in the next test down the chain. This means I can run each part of the chain or even a length of the chain at any time and I don't have to pay the cost of having to set up all the dependencies all the time.

Long explanation: The problem with Integration Tests and dependencies is that they kill you slowly but surely. There are several reasons for this. The main ones are:

If you have a test that executes a million lines or code and it fails, then you have to find the problem in a million lines of code. No matter how smart/good you are, this will take more time than looking for a problem is a dozen lines of code. Plus any change in any of those lines might break the test. Which means you'll break the test a lot because it depends on so much code that it's impossible to change anything without breaking tests ... or your test doesn't test anything for which the technical term is "waste of time".

The second reason is that the human brain is simply too small to keep all the details of those dependencies at hand.

Lastly, because you have to set up so much, the tests will be slow. A lot of code will be executed thousands of times without giving you much benefit. Or they won't be executed a lot of times because no one has time to wait an hour for all the tests to complete and people eventually starts to abandon the tests. Rule of thumb: You shouldn't wait for than 10 seconds on the tests while making changes to the code.

After struggling with this for a couple of years, I eventually came to revisit something my first boss told me: Software is predictable. If it's not, then the software is right - the model of the software in your head is wrong. He also said that no matter how complex a software is, you can always break it down to:

input -> processing -> output.

It's self-similar like a fractal: If "processing" is complex, that just means it's a chain of several i-p-o elements. So your example


means: input -> C1 -> A -> C2 -> B -> C3 -> output

which can be broken down into:

input -> C1 -> output-of-C a.k.a input-for-A
input-for-A -> A -> output-A-result-for-C2
output-A-result-for-C2 -> C2 -> input-for-B
input-for-B -> B -> result-for-C3
result-for-C3 -> C3 -> output-of-C

You have two options to implement this:
  • create mocks of A and B which test "is the input what the test expects" and then return the canned result.
  • Split C into three smaller pieces so you can get test the pieces directly without mocks.
  • Tests for A and B can go into the sets for those features.
  • Most IDEs can show you where something is used. So if you make changes to A, you can search which other tests use the canned inputs and outputs and see who is affected.
  • While you work on A, you see only the tests for A break. That means you can focus on one thing at a time.
  • It makes you cut features into smaller pieces which can be tested effectively.
  • Eventually, breaking changes will stop propagating through all the code. This kind of development is like setting up firewalls.
  • It's roughly the same amount of work, it just needs a mental shift.
Example: Login form. The login form has many paths (new user, known user, lost password) but the next step doesn't care: it just expects a valid User object.

Post Reply

Return to “News”