Bugs in 0.2.9 and testing

Information about releases and roadmap.
Post Reply
slpwnd
Factorio Staff
Factorio Staff
Posts: 1835
Joined: Sun Feb 03, 2013 2:51 pm
Contact:

Bugs in 0.2.9 and testing

Post by slpwnd »

There were some serious bugs reported in 0.2.9 release in our bugs section (labs not being mineable, crashes in research, errors in crafting / smelting, etc.). We are very sorry for these. It is very painful for us to see that you guys are having issues with Factorio and are frustrated from the bugs.

The problem is on our side. We have to admit that we did the release in a rush in order to keep our release schedule (uptil now we did a release every Friday). Also we brought in some core rewrites that haven't been properly tested. Hastiness brings errors. We learned our lesson. We will definitely be more careful next time.

As for the current situation we will not be able to make a patch release today. We might do one tomorrow (sunday) though. Another option is taking 0.2.9 down completely. There is always option to download previous release from our download page (this time 0.2.8).

It is quite difficult for us to do proper testing these days though. We are just two guys and there are so many things to do. Therefore we are actually considering asking the community for help. The way we picture this could work is as follows:
1) We might setup couple of "early tester" accounts on our website for people interested in this
2) We would create the release in advance (wednesday / thursday or so) and freeze it (no more new code coming in - develop in the branch).
3) We would then notify the early testers and they could download the new release.
4) The testers would report us any serious bugs they find and we would fix those before doing the regular (friday evening) release.

This might be a bit of overkill. We would be happy to know your opinion.

Our apologies for the issues with the 0.2.9 release again.

Balinor
Fast Inserter
Fast Inserter
Posts: 147
Joined: Tue Feb 19, 2013 12:35 pm
Contact:

Re: Bugs in 0.2.9 and testing

Post by Balinor »

Hi guys.

I think all developers fall into the trap of wanting people to see what new shiny they have been working on and end up rushing themselves. Don't beat yourselves up too much, we've all been there.

Personally I'd keep two versions on the website, the most recent stable version and the most recent version that requires testing. Once the test version has been put through the ringer by the community and you have received no notifications of game breaking bugs, it's time to move it to stable ready for a new test version. That in my experience has been a good way to deal with alphas and testing. Of course, your mileage may vary.

On the issue of the paypal problems, do you have a timeframe for when you are expecting to hear more information? I am currently sitting on some extra money that I think deserves a place with you guys.

Regards.

*EDIT*
One thing I forgot to mention with this approach. It is usual to just include the test as either a zipfile, or a seperate test install. As you do not currently use a system for automatic updates, I would personally just include the latest version as a zipfile and only build new install packages for when a version is marked as stable. This may have the added benefit of saving you guys a little bit of time also.
Last edited by Balinor on Sat Mar 02, 2013 3:41 pm, edited 1 time in total.

MF-
Smart Inserter
Smart Inserter
Posts: 1235
Joined: Sun Feb 24, 2013 12:07 am
Contact:

Re: Bugs in 0.2.9 and testing

Post by MF- »

Good idea.
Simply put a collapsible section on the top marked "testing".
And access to the section would require saying "Okay" on a message saying something like "This has not been tested, it may delete your saves, vaporize your computer or make your dog grow tree more heads !!"

and mark the rest of the builds "(reasonably) stable"

Milaha
Inserter
Inserter
Posts: 23
Joined: Thu Feb 21, 2013 11:52 pm
Contact:

Re: Bugs in 0.2.9 and testing

Post by Milaha »

This is alpha, we should know there are going to be bugs, and lots of them. I do not think you need "early testers" we are all early testers. I do agree that simply marking a known to be solid build as stable is a good idea though. Likely keeping the demo on that version until a new one happens by.

User avatar
Ween
Long Handed Inserter
Long Handed Inserter
Posts: 62
Joined: Sun Feb 17, 2013 1:53 pm
Contact:

Re: Bugs in 0.2.9 and testing

Post by Ween »

We know the game is still in development. We know you are doing your best !!!
---<@

Xisuma
Inserter
Inserter
Posts: 29
Joined: Fri Mar 01, 2013 1:33 pm
Contact:

Re: Bugs in 0.2.9 and testing

Post by Xisuma »

Update the game when you think its ready, dont push yourselfs! :-)

orbito
Long Handed Inserter
Long Handed Inserter
Posts: 57
Joined: Sat Feb 23, 2013 7:19 pm
Contact:

Re: Bugs in 0.2.9 and testing

Post by orbito »

I would be happy to help with early alpha testing. How do I sign up?

Note: I have five years experience writing bugs and running test cases.

User avatar
Ween
Long Handed Inserter
Long Handed Inserter
Posts: 62
Joined: Sun Feb 17, 2013 1:53 pm
Contact:

Re: Bugs in 0.2.9 and testing

Post by Ween »

orbito wrote:I would be happy to help with early alpha testing. How do I sign up?

Note: I have five years experience writing bugs and running test cases.
Same here :p
---<@

HellEye
Inserter
Inserter
Posts: 37
Joined: Thu Feb 21, 2013 9:37 pm
Contact:

Re: Bugs in 0.2.9 and testing

Post by HellEye »

I'll be more than happy to help you guys in testing factorio. As to updates: I think that making release every friday is overkill, not asking us to help. You should have one stable relase, like 0.2.8 and maybe twice a week send testers fresh code with all of bugs etc, so they cab check it, and name it maybe 0.2.8a, then b and so on. If one of beta realses won't get milions of hard bugs it can become 0.2.9 in two days. For me it will be pretty good update system.

LoSboccacc
Filter Inserter
Filter Inserter
Posts: 251
Joined: Wed Feb 20, 2013 9:35 pm
Contact:

Re: Bugs in 0.2.9 and testing

Post by LoSboccacc »

Personally I hate when devs make first class and second class citizen by creating a restricted test group.
But I am fine with a bleeding edge version as long as it is open to everyone.A word of warning however: for a small team will become extremely time consuming to process the feedback from a test version.

slpwnd
Factorio Staff
Factorio Staff
Posts: 1835
Joined: Sun Feb 03, 2013 2:51 pm
Contact:

Re: Bugs in 0.2.9 and testing

Post by slpwnd »

Thank you all for the input regarding the idea of having a testing release. In the end we decided to go with the following:

We will have one release marked as stable (at the moment 0.2.8) and one release marked as experimental (at the moment upcoming 0.2.10 which fixes horrible bugs from 0.2.9). On the website the default will be the stable one, but all the users will be allowed to download the experimental as well. There will also be warning now that "experimental" means possibly serious bugs. Once it is obvious that the "experimental" release doesn't contain any obvious / serious bugs (based on user reports from the forum) we will mark it as stable. The changes on the website will apply since tomorrow after the Indiegogo campaign is finished (there is a whole pack of update and I don't want to touch the site now).

Also once we manage to get more or less stable version (high hopes for 0.2.10) we will be doing releases with way less frequency. Basically the release scheme a release every week that we tried to perform was a silly thing to do (especially in two people who simultaneously do 10 other things). As you guys said: quality over quantity.

Dakkanor
Fast Inserter
Fast Inserter
Posts: 243
Joined: Wed Feb 27, 2013 6:18 am
Contact:

Re: Bugs in 0.2.9 and testing

Post by Dakkanor »

LoSboccacc wrote:Personally I hate when devs make first class and second class citizen by creating a restricted test group.
But I am fine with a bleeding edge version as long as it is open to everyone.A word of warning however: for a small team will become extremely time consuming to process the feedback from a test version.
how about bugzilla? i know that the gnomoria creator uses it

or possible just have someone compile a list of bugs from the forums that come up with each build

on the note of updates, the community would probably be okay with weeks skipped provided you warned them about it before hand (i.e. you realize you might not make a stable build for a week and maybe post an unstable release for anyone who wants to try it anyway

orbito
Long Handed Inserter
Long Handed Inserter
Posts: 57
Joined: Sat Feb 23, 2013 7:19 pm
Contact:

Re: Bugs in 0.2.9 and testing

Post by orbito »

Yes, this game really should be using a database like Bugzilla.

There should also be a "test release" available to us before each official release. That way, people who want to test can do so and people who want to play a stable build can stick with their older version until the official release. It's the perfect solution.

Angelic
Inserter
Inserter
Posts: 24
Joined: Mon Feb 18, 2013 4:00 pm
Contact:

Re: Bugs in 0.2.9 and testing

Post by Angelic »

kovarex posted a link to a bug-list (which I believe was editable by everyone), but I'll be damned if I can find it -.-

slpwnd
Factorio Staff
Factorio Staff
Posts: 1835
Joined: Sun Feb 03, 2013 2:51 pm
Contact:

Re: Bugs in 0.2.9 and testing

Post by slpwnd »

orbito wrote:Yes, this game really should be using a database like Bugzilla.
We will most probably want a bugtracker. We considered using the Github built-in one originally.
orbito wrote:There should also be a "test release" available to us before each official release. That way, people who want to test can do so and people who want to play a stable build can stick with their older version until the official release. It's the perfect solution.
We have introduced the "stable" - "experimental" release concept. Now on the download page at the website you can choose whether you want the stable (default - 0.2.8) or experimental (bugs warning - 0.2.10) release.

MF-
Smart Inserter
Smart Inserter
Posts: 1235
Joined: Sun Feb 24, 2013 12:07 am
Contact:

Re: Bugs in 0.2.9 and testing

Post by MF- »

That "experimental release" link looks confusing.
I would expect text "Stable, tested release" exactly in there.

Post Reply

Return to “Releases”