Friday Facts #186 - Marathon testing

Regular reports on Factorio development.
ealrann
Burner Inserter
Burner Inserter
Posts: 10
Joined: Sat Aug 06, 2016 8:04 pm
Contact:

Re: Friday Facts #186 - Marathon testing

Post by ealrann »

Ohz wrote:
ealrann wrote:Nah, don't give release date. It's definitely a bad idea (too stressful for a dev team, source of bad coding and more).

However, a good practice is too make frequent builds (I think to Agile Manifesto). So, why don't you give us some kind of Steam Beta Code in the forum, with a big warning : "These builds will crash or corrupt your save". So we could access some staging builds and satisfy our thirst :) We definitely want to be yours beta/alpha testers :D, it's ok if the game crash or if you corrupt our save (we would have been warned).

There were a terrible bug when they released 0.13.1 or something that when you place concrete and remove it, it creates water. Lot of people played their saves and destroyed their factories because of it. Yep, now we can remove water, but it was about a year later (they fixed a bug in a day, but the water remains, you had to dl a mod to fix it). Yep, the warning was there, but still 1) people got angry, and 2) well you don't really want to play with a so broken feature.

I definitely believe they learnt from this day, we never had a massive break-gaming bug since this day.
That's why the beta code I propose should only be accessible in the forum, with a big warning.

suprnova74
Long Handed Inserter
Long Handed Inserter
Posts: 61
Joined: Thu Sep 10, 2015 2:29 am
Contact:

Re: Friday Facts #186 - Marathon testing

Post by suprnova74 »

Working in a job where i have to code myself, i can totally understand how bug-hunting can be a necessary but time-consuming affair. Many times I will think something will take x time and hunting that elusive bug, OS quirk, rare condition, etc makes it take x*10. It's seems to follow the Pareto-80/20 rule. 80% of the functionality is written in 20% of the time. 20% of the remaining items feels like it takes 80% of the time (including bug hunting).

Just remember that the people grumbling about it taking too long are merely excited for the new features. You got them hooked :)

aober93
Filter Inserter
Filter Inserter
Posts: 453
Joined: Tue Aug 30, 2016 9:07 pm
Contact:

Re: Friday Facts #186 - Marathon testing

Post by aober93 »

ili wrote:
Ohz wrote:So what?
It's an Experimental release.
Its a commercial game and with this release being advertised there is nothing experimental about this in terms of legal , customer service and customer satisfaction. The devs testing the game before release is just that.

biliskner25
Burner Inserter
Burner Inserter
Posts: 18
Joined: Mon Dec 08, 2014 8:18 am
Contact:

Re: Friday Facts #186 - Marathon testing

Post by biliskner25 »

Image

HammerPiano
Fast Inserter
Fast Inserter
Posts: 232
Joined: Thu Dec 31, 2015 7:36 am
Contact:

Re: Friday Facts #186 - Marathon testing

Post by HammerPiano »

V453000 wrote:Just bear with us for a very short while please, we are doing literally everything we can to make it happen as soon as possible.

I'm not going to cry about how much time we spend, how much effort we put, how close it can get to taxing on both mental and physical health and whatever. We took a big bit of a release to chew this time on, but when you get it, I believe you will trust me that the wait was worth it.

We can already see the light at the end of the LONG tunnel, and the light is very bright. : )

Thank you

V
<3
Your game is awesome, take your time to make it more awesome !!!

Engimage
Smart Inserter
Smart Inserter
Posts: 1067
Joined: Wed Jun 29, 2016 10:02 am
Contact:

Re: Friday Facts #186 - Marathon testing

Post by Engimage »

V453000 wrote:Just bear with us for a very short while please, we are doing literally everything we can to make it happen as soon as possible.

I'm not going to cry about how much time we spend, how much effort we put, how close it can get to taxing on both mental and physical health and whatever. We took a big bit of a release to chew this time on, but when you get it, I believe you will trust me that the wait was worth it.

We can already see the light at the end of the LONG tunnel, and the light is very bright. : )

Thank you

V
We are with you guys. You are awesome. But we are really tired of a journey on this overhyped train.

aka13
Filter Inserter
Filter Inserter
Posts: 671
Joined: Sun Sep 29, 2013 1:18 pm
Contact:

Re: Friday Facts #186 - Marathon testing

Post by aka13 »

Now whz I specifically love factorio is because of your attitude towards the playerbase. Even if it says "early-access" on steam, every second of factorio feels like sitting on mission-critical software with 0 bugs. As much as I would love to have 0.15 "as it is" right now, I really do appreciate that every release, even the experimental ones feel solid and bugless. Take your time, devs, we all love you and appreciate your effort. Also, it is such a relief seeing the devs play their own game, knowing that balancing and gameplay will be thought out and not just a pack of ideas. 0.15 is going to be amazing, as every release.
Pony/Furfag avatar? Opinion discarded.

User avatar
brunzenstein
Smart Inserter
Smart Inserter
Posts: 1059
Joined: Tue Mar 01, 2016 2:27 pm
Contact:

Re: Friday Facts #186 - Marathon testing

Post by brunzenstein »

aka13 wrote:Now whz I specifically love factorio is because of your attitude towards the playerbase. Even if it says "early-access" on steam, every second of factorio feels like sitting on mission-critical software with 0 bugs. As much as I would love to have 0.15 "as it is" right now, I really do appreciate that every release, even the experimental ones feel solid and bugless. Take your time, devs, we all love you and appreciate your effort. Also, it is such a relief seeing the devs play their own game, knowing that balancing and gameplay will be thought out and not just a pack of ideas. 0.15 is going to be amazing, as every release.
I sign that statement in full

Albrat
Inserter
Inserter
Posts: 34
Joined: Tue Feb 17, 2015 2:35 pm
Contact:

Re: Friday Facts #186 - Marathon testing

Post by Albrat »

suprnova74 wrote:Working in a job where i have to code myself, i can totally understand how bug-hunting can be a necessary but time-consuming affair. Many times I will think something will take x time and hunting that elusive bug, OS quirk, rare condition, etc makes it take x*10. It's seems to follow the Pareto-80/20 rule. 80% of the functionality is written in 20% of the time. 20% of the remaining items feels like it takes 80% of the time (including bug hunting).

Just remember that the people grumbling about it taking too long are merely excited for the new features. You got them hooked :)
usually at least 10 to the power times longer than you expect.

I am glad we are getting 0.15 , I am really excited for the new version. But I like to play the game as it is as well... I have a version 0.10.x install , 0.11.15 install, 0.12.35 install, 0.13.21 install and a 0.14.22 with mods install. My current videos are made on the 13.21 install of factorio still. I am planning out my 14.22 mods playlist already, but I am thinking of jumping into a 0.15.x series as soon as it is released. I am really looking forward to this release.

BUT!!! I am willing to wait as long as it takes for the development team to be perfectly happy with the 0.15.x version before they release it. I have been part of a lot of early access and Alpha / beta games. (and operating systems) The development process takes years and can result in no actual product available for release at the end of it as well. (been part of alpha's that got abandonned before they even got to beta versions. NB: abandonned some of my own ideas in Alpha as my skills were not good enough to do the game idea justice.)
I am glad that factorio is developing as fast as it is. We sure get alot of game and gameplay for the money we have paid.

mattj256
Fast Inserter
Fast Inserter
Posts: 203
Joined: Sun Mar 27, 2016 7:25 am
Contact:

Re: Friday Facts #186 - Marathon testing

Post by mattj256 »

For the following release (0.16), can you make it a smaller release and more quickly?
It's so hard to wait, especially when no one knows exactly how long it will take!
(Keep up the good work.)

User avatar
commandoby
Inserter
Inserter
Posts: 32
Joined: Wed Apr 27, 2016 1:37 pm
Contact:

Re: Friday Facts #186 - Marathon testing

Post by commandoby »

mattj256 wrote:For the following release (0.16), can you make it a smaller release and more quickly?
It's so hard to wait, especially when no one knows exactly how long it will take!
(Keep up the good work.)
The best way to wait is to get distracted by something else and stop waiting. =)

BenSeidel
Filter Inserter
Filter Inserter
Posts: 584
Joined: Tue Jun 28, 2016 1:44 am
Contact:

Re: Friday Facts #186 - Marathon testing

Post by BenSeidel »

commandoby wrote:The best way to wait is to get distracted by something else and stop waiting. =)
Unfortunately there is no other game that has captured my attention like Factorio has. Having tried MANY other games, the well over 3000+ hours that I have sunk into this game and I STILL WANT MORE is testament to the massive chasm that separates it from anything else. As I really don't play a map for anything less than 100 hours, this week by week waiting is just destroying me!

Vinnie_NL
Inserter
Inserter
Posts: 39
Joined: Tue Mar 01, 2016 8:03 pm
Contact:

Re: Friday Facts #186 - Marathon testing

Post by Vinnie_NL »

To everyone here asking for a release date, I sure feel the same but I think it's for the best for the game and all of us if we just let the devs decide when it's ready. Our job is to enjoy the game and suggest possible game improvements, everything else is for the devs. I think we get the best possible game that way, so that should make waiting easier.

I remember reading a comment here stating that this shouldn't be a new Duke Nukem Forever done when it's done situation. But the development of Factorio can't be compared to that *ahum* game. With Duke Nukem we were teased for years that it would be the best game ever, with only sporadically releasing a new trailer but no detailed information whatsoever about the status of the game, over a period of 16 (!) years. In 2011 we get the game, and it didn't wasn't the classic everyone was hoping for. I think everyone has forgotten about it by now.

I haven't been in that many early access games, but I think it's safe to say that the Factorio development team is one of the most open developers there is. They share a new blog every week with not only interesting game info, but also the challenges and the reasons why they choose this release schedule. Maybe there are other companies having a better release schedule for their situation, but they don't have Factorio, the best industrial automation simulation you can think of :D

deasturies
Manual Inserter
Manual Inserter
Posts: 2
Joined: Sat Nov 19, 2016 7:48 am
Contact:

Re: Friday Facts #186 - Marathon testing

Post by deasturies »

Regarding automated testing: how it is organised from technical point of view ? What technology stack ? How do you assert that all is good ? Does it involve image capture and comparing to a benchmark ?

Twinsen
Factorio Staff
Factorio Staff
Posts: 1329
Joined: Tue Sep 23, 2014 7:10 am
Contact:

Re: Friday Facts #186 - Marathon testing

Post by Twinsen »

deasturies wrote:Regarding automated testing: how it is organised from technical point of view ? What technology stack ? How do you assert that all is good ? Does it involve image capture and comparing to a benchmark ?
We write tests in C++ code. We place some entities using code, set them up as we want and then check that they end up in the correct state.
Normally tests run without graphical mode. Entities are created but nothing is ever rendered. We sometimes use graphical mode to see how the test is running and if we placed the entities correctly.

Here's how one of our simpler tests looks like:

Code: Select all

  TEST(AccumulatorCircuitNetworkReadChargeLevel)
  {
    TestScenarioHolder scenario;
    scenario.createArea(10);

    //set accumulator to different charge levels and test if the circuit network output sends the correct precentage
    Accumulator* accumulator = scenario.addEntity<Accumulator>("accumulator", RealPosition(2, 2));
    accumulator->setAvailableEnergy(accumulator->electricEnergySource->getBufferSize());
    ElectricPole* pole = scenario.addEntity<ElectricPole>("medium-electric-pole", RealPosition(4.5, 2.5));
    WireConnectionDefinition other = WireConnectionDefinition(pole, ElectricPole::circuitConnectorID);
    accumulator->getOrCreateCircuitConnector(AccumulatorControlBehavior::circuitConnectorID)->connectTo(&other, Wire::Type::Green);
    dynamic_cast<AccumulatorControlBehavior*>(accumulator->getOrCreateControlBehavior())->outputSignalId = VirtualSignalPrototypeList::get("signal-A")->getID();

    scenario.update();

    CircuitNetwork* network = pole->getOrCreateCircuitConnector(ElectricPole::circuitConnectorID)->getGreenNetwork();
    CHECK_EQUAL(uint32_t(network->getSignalValues().size()), 1u);
    CHECK_EQUAL(network->getValue(VirtualSignalPrototypeList::get("signal-A")->getID()), 100);
    scenario.update();

    accumulator->setAvailableEnergy(0);
    scenario.update();

    CHECK_EQUAL(uint32_t(network->getSignalValues().size()), 0u);
    scenario.update();

    accumulator->setAvailableEnergy(accumulator->electricEnergySource->getBufferSize() / 2);
    scenario.update();

    CHECK_EQUAL(uint32_t(network->getSignalValues().size()), 1u);
    CHECK_EQUAL(network->getValue(VirtualSignalPrototypeList::get("signal-A")->getID()), 50);
    scenario.update();
}

biliskner25
Burner Inserter
Burner Inserter
Posts: 18
Joined: Mon Dec 08, 2014 8:18 am
Contact:

Re: Friday Facts #186 - Marathon testing

Post by biliskner25 »

Twinsen wrote: We write tests in C++ code. We place some entities using code, set them up as we want and then check that they end up in the correct state.
Normally tests run without graphical mode. Entities are created but nothing is ever rendered. We sometimes use graphical mode to see how the test is running and if we placed the entities correctly.
Factorioception

Image

KingBenjai
Manual Inserter
Manual Inserter
Posts: 2
Joined: Thu Jan 05, 2017 1:32 am
Contact:

Re: Friday Facts #186 - Marathon testing

Post by KingBenjai »

The way I've been dealing with these delays is throwing myself into bob/angels petrochem multiple times.

liij
Burner Inserter
Burner Inserter
Posts: 15
Joined: Sun Aug 17, 2014 8:39 am
Contact:

Re: Friday Facts #186 - Marathon testing

Post by liij »

Please 0.15 tomorrow ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;..............................................; ;..........................................................................; ;........................; ;..................................; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.; ;.;

Mercury044
Burner Inserter
Burner Inserter
Posts: 16
Joined: Tue Aug 04, 2015 8:21 pm
Contact:

Re: Friday Facts #186 - Marathon testing

Post by Mercury044 »

I want to say thank you to the Dev team for making this great game.

i can wait for 0.15 but don't want to as this game has takes so much of my life and yet i still want more. You have created a masterpiece of a game and I love it.

Don't hurt yourselves but please feed my addiction soonest.

Thanks again!

drmason13
Inserter
Inserter
Posts: 33
Joined: Sat Apr 16, 2016 1:47 pm
Contact:

Re: Friday Facts #186 - Marathon testing

Post by drmason13 »

A word of encouragement for the devs.

Every post complaining about the progress of this release is a fan who really really likes your game! :)

In fact I have a theory.
The increase in complaints is caused by "factorio withdrawal". A lot of us are trying really hard NOT to play your game so we are fresh and eager for a binge come 0.15. The problem is your game is too good, so not playing it makes us cranky.

I just started a new game yesterday. I'm not putting any pressure on it. I'll quit it when 0.15 arrives. Boy do I feel better for it :)

On a side note: trees are a pain at the start of the game aren't they? I have an idea for it in this post here:

viewtopic.php?f=6&t=44254
Last edited by drmason13 on Mon Apr 17, 2017 7:13 am, edited 1 time in total.

Post Reply

Return to “News”