Bugs in 0.2.9 and testing

[slpwnd]

There were some serious bugs reported in 0.2.9 release in our bugs section (labs not being mineable, crashes in research, errors in crafting / smelting, etc.). We are very sorry for these. It is very painful for us to see that you guys are having issues with Factorio and are frustrated from the bugs.

The problem is on our side. We have to admit that we did the release in a rush in order to keep our release schedule (uptil now we did a release every Friday). Also we brought in some core rewrites that haven't been properly tested. Hastiness brings errors. We learned our lesson. We will definitely be more careful next time.

As for the current situation we will not be able to make a patch release today. We might do one tomorrow (sunday) though. Another option is taking 0.2.9 down completely. There is always option to download previous release from our download page (this time 0.2.8).

It is quite difficult for us to do proper testing these days though. We are just two guys and there are so many things to do. Therefore we are actually considering asking the community for help. The way we picture this could work is as follows:
1) We might setup couple of "early tester" accounts on our website for people interested in this
2) We would create the release in advance (wednesday / thursday or so) and freeze it (no more new code coming in - develop in the branch).
3) We would then notify the early testers and they could download the new release.
4) The testers would report us any serious bugs they find and we would fix those before doing the regular (friday evening) release.

This might be a bit of overkill. We would be happy to know your opinion.

Our apologies for the issues with the 0.2.9 release again.

[Balinor]

Hi guys.

I think all developers fall into the trap of wanting people to see what new shiny they have been working on and end up rushing themselves. Don't beat yourselves up too much, we've all been there.

Personally I'd keep two versions on the website, the most recent stable version and the most recent version that requires testing. Once the test version has been put through the ringer by the community and you have received no notifications of game breaking bugs, it's time to move it to stable ready for a new test version. That in my experience has been a good way to deal with alphas and testing. Of course, your mileage may vary.

On the issue of the paypal problems, do you have a timeframe for when you are expecting to hear more information? I am currently sitting on some extra money that I think deserves a place with you guys.

Regards.

*EDIT*
One thing I forgot to mention with this approach. It is usual to just include the test as either a zipfile, or a seperate test install. As you do not currently use a system for automatic updates, I would personally just include the latest version as a zipfile and only build new install packages for when a version is marked as stable. This may have the added benefit of saving you guys a little bit of time also.

[MF-]

Good idea.
Simply put a collapsible section on the top marked "testing".
And access to the section would require saying "Okay" on a message saying something like "This has not been tested, it may delete your saves, vaporize your computer or make your dog grow tree more heads !!"

and mark the rest of the builds "(reasonably) stable"

[Milaha]

This is alpha, we should know there are going to be bugs, and lots of them. I do not think you need "early testers" we are all early testers. I do agree that simply marking a known to be solid build as stable is a good idea though. Likely keeping the demo on that version until a new one happens by.

[Ween]

We know the game is still in development. We know you are doing your best !!!

[Xisuma]

Update the game when you think its ready, dont push yourselfs!

[orbito]

I would be happy to help with early alpha testing. How do I sign up?

Note: I have five years experience writing bugs and running test cases.

[Ween]

I would be happy to help with early alpha testing. How do I sign up?

Note: I have five years experience writing bugs and running test cases.

Same here :p

[HellEye]

I'll be more than happy to help you guys in testing factorio. As to updates: I think that making release every friday is overkill, not asking us to help. You should have one stable relase, like 0.2.8 and maybe twice a week send testers fresh code with all of bugs etc, so they cab check it, and name it maybe 0.2.8a, then b and so on. If one of beta realses won't get milions of hard bugs it can become 0.2.9 in two days. For me it will be pretty good update system.

[LoSboccacc]

Personally I hate when devs make first class and second class citizen by creating a restricted test group.
But I am fine with a bleeding edge version as long as it is open to everyone.A word of warning however: for a small team will become extremely time consuming to process the feedback from a test version.

[slpwnd]

Thank you all for the input regarding the idea of having a testing release. In the end we decided to go with the following:

We will have one release marked as stable (at the moment 0.2.8) and one release marked as experimental (at the moment upcoming 0.2.10 which fixes horrible bugs from 0.2.9). On the website the default will be the stable one, but all the users will be allowed to download the experimental as well. There will also be warning now that "experimental" means possibly serious bugs. Once it is obvious that the "experimental" release doesn't contain any obvious / serious bugs (based on user reports from the forum) we will mark it as stable. The changes on the website will apply since tomorrow after the Indiegogo campaign is finished (there is a whole pack of update and I don't want to touch the site now).

Also once we manage to get more or less stable version (high hopes for 0.2.10) we will be doing releases with way less frequency. Basically the release scheme a release every week that we tried to perform was a silly thing to do (especially in two people who simultaneously do 10 other things). As you guys said: quality over quantity.

[Dakkanor]

Personally I hate when devs make first class and second class citizen by creating a restricted test group.
But I am fine with a bleeding edge version as long as it is open to everyone.A word of warning however: for a small team will become extremely time consuming to process the feedback from a test version.

how about bugzilla? i know that the gnomoria creator uses it

or possible just have someone compile a list of bugs from the forums that come up with each build

on the note of updates, the community would probably be okay with weeks skipped provided you warned them about it before hand (i.e. you realize you might not make a stable build for a week and maybe post an unstable release for anyone who wants to try it anyway

[orbito]

Yes, this game really should be using a database like Bugzilla.

There should also be a "test release" available to us before each official release. That way, people who want to test can do so and people who want to play a stable build can stick with their older version until the official release. It's the perfect solution.

[Angelic]

kovarex posted a link to a bug-list (which I believe was editable by everyone), but I'll be damned if I can find it -.-

[slpwnd]

Yes, this game really should be using a database like Bugzilla.

We will most probably want a bugtracker. We considered using the Github built-in one originally.

There should also be a "test release" available to us before each official release. That way, people who want to test can do so and people who want to play a stable build can stick with their older version until the official release. It's the perfect solution.

We have introduced the "stable" - "experimental" release concept. Now on the download page at the website you can choose whether you want the stable (default - 0.2.8) or experimental (bugs warning - 0.2.10) release.

[MF-]

That "experimental release" link looks confusing.
I would expect text "Stable, tested release" exactly in there.