Aardvark Bellay
14.12.02, 22:11
Heute vom producer(chantz) im Infogrames Forum gepostet:
Folks.
Some people are confused about how regression testing works. I'm going to take minute to hopefully give you some insight into why it takes so long to test moo3. Note that companies have varying methodology but the end result is the *same*; shake out all the major bugs.
1: MOO3 is huge. HUGE. Did i say how huge it is? There are so many permutations and so many ways to "skin a cat" that testing them all takes a team of 20 people working roughly 16 hours a day 3 straight weeks to slog through it. That's with people assigned to specific areas of focus. Did i say how huge it is? IT'S HUGE.
2: Re: Regression testing. See the above statement? Every time we get a new build, we start from scratch and test every known permutation. Bugs come up, they always do. The important thing to note here is that *are the bugs that come up during a regression test show stoppers and the ones that we're marked as showstoppers in the previous build fixed?*
3: What's a showstopper? Many, many things. A repeatable crash. Crashing after 2-4-6 hours of gameplay. Running out of memory and crashing. Save game corruption. Multiplayer hangs. Missing feature or a feature not working properly. Missing art; the list goes on and on. We have an enormous checklist we go through. Every time we fix a bug, we take a chance that we might break something else. To mitigate this, especially during the final weeks of testing, we enter into extreme paranoia mode. We call this Code Review. Every time a programmer makes a change, it is reviewed by at least 2 other programmers who share some knowledge of that particular piece of code. A line-by-line review is done, questions are asked, flags are raised (or not, as the case may be). That bug fix then gets tested for at least a few hours before it gets "checked in" to the main source tree. It gets compiled into the build (along with any other fixes) and is then sent off for proper (see above) testing.
4: Source code control. Know it, love it, it is your best friend. When a programmer makes a change to a source or header file (or creates a new one), it is done outside the main "tree", which resides on a SERVER that all team members have access to. In other words, it is done locally on that programmer’s machine. So, if i create a file or modify a file called spies.cpp (c++ extension), i'd do it locally to ensure that i'm not stomping on code in the main tree and causing dumb errors and stopping other people from working. A file can only be checked out by one programmer (or designer or artist for that matter) at a time, for obvious reasons. This is really the condensed version, as there is much more to it, but this sums it up in a nutshell.
5: Sometimes we know right away that a build doesn't cut the mustard (a crash 20 turns into it), but we are *now* at the point where most problems are occurring very LATE GAME in MP on Large maps with many races; in other words, we are exercising and stress testing areas of the game we couldn't 6 weeks ago because we couldn't GET there due to crashes and errors. Game balance also figured into this. Heck, there was a time when you could *not* win against the new orions no matter how many ships you could throw at them (turn after turn). That means no Sole Superpower Victory. That's a Showstopper because you cannot win the game - at all. No fun.
***So, basically, every time we get a new gold candidate build we put it through rigorous testing (see above), weigh the pros and cons and DECIDE whether it is good enough to go gold. Right now, it is not. And that is our decision to make an no one else’s.***
Lastly, I *did* make a mistake telling you folks that it would go gold Nov 26th. I honestly thought it would because THAT WAS THE DIRECTIVE from above. Period, no matter what, it goes gold the 26th. That changed for reasons i cannot and will not go into. Am i happy that i got the extra time? YOU BET YOUR BOOTY I AM because the game will only be better for it. Am i bummed it isn't out already? You bet i am, because no one wants it finished, out the door and into your hands more than me and the folks at QSI. Catch 22 eh?
Anyway, hopefully this gives you some insight into how regression testing (at least in my little world) works. No questions please, just enjoy (or not, as the case may be).
Thanks,
Constantine
Folks.
Some people are confused about how regression testing works. I'm going to take minute to hopefully give you some insight into why it takes so long to test moo3. Note that companies have varying methodology but the end result is the *same*; shake out all the major bugs.
1: MOO3 is huge. HUGE. Did i say how huge it is? There are so many permutations and so many ways to "skin a cat" that testing them all takes a team of 20 people working roughly 16 hours a day 3 straight weeks to slog through it. That's with people assigned to specific areas of focus. Did i say how huge it is? IT'S HUGE.
2: Re: Regression testing. See the above statement? Every time we get a new build, we start from scratch and test every known permutation. Bugs come up, they always do. The important thing to note here is that *are the bugs that come up during a regression test show stoppers and the ones that we're marked as showstoppers in the previous build fixed?*
3: What's a showstopper? Many, many things. A repeatable crash. Crashing after 2-4-6 hours of gameplay. Running out of memory and crashing. Save game corruption. Multiplayer hangs. Missing feature or a feature not working properly. Missing art; the list goes on and on. We have an enormous checklist we go through. Every time we fix a bug, we take a chance that we might break something else. To mitigate this, especially during the final weeks of testing, we enter into extreme paranoia mode. We call this Code Review. Every time a programmer makes a change, it is reviewed by at least 2 other programmers who share some knowledge of that particular piece of code. A line-by-line review is done, questions are asked, flags are raised (or not, as the case may be). That bug fix then gets tested for at least a few hours before it gets "checked in" to the main source tree. It gets compiled into the build (along with any other fixes) and is then sent off for proper (see above) testing.
4: Source code control. Know it, love it, it is your best friend. When a programmer makes a change to a source or header file (or creates a new one), it is done outside the main "tree", which resides on a SERVER that all team members have access to. In other words, it is done locally on that programmer’s machine. So, if i create a file or modify a file called spies.cpp (c++ extension), i'd do it locally to ensure that i'm not stomping on code in the main tree and causing dumb errors and stopping other people from working. A file can only be checked out by one programmer (or designer or artist for that matter) at a time, for obvious reasons. This is really the condensed version, as there is much more to it, but this sums it up in a nutshell.
5: Sometimes we know right away that a build doesn't cut the mustard (a crash 20 turns into it), but we are *now* at the point where most problems are occurring very LATE GAME in MP on Large maps with many races; in other words, we are exercising and stress testing areas of the game we couldn't 6 weeks ago because we couldn't GET there due to crashes and errors. Game balance also figured into this. Heck, there was a time when you could *not* win against the new orions no matter how many ships you could throw at them (turn after turn). That means no Sole Superpower Victory. That's a Showstopper because you cannot win the game - at all. No fun.
***So, basically, every time we get a new gold candidate build we put it through rigorous testing (see above), weigh the pros and cons and DECIDE whether it is good enough to go gold. Right now, it is not. And that is our decision to make an no one else’s.***
Lastly, I *did* make a mistake telling you folks that it would go gold Nov 26th. I honestly thought it would because THAT WAS THE DIRECTIVE from above. Period, no matter what, it goes gold the 26th. That changed for reasons i cannot and will not go into. Am i happy that i got the extra time? YOU BET YOUR BOOTY I AM because the game will only be better for it. Am i bummed it isn't out already? You bet i am, because no one wants it finished, out the door and into your hands more than me and the folks at QSI. Catch 22 eh?
Anyway, hopefully this gives you some insight into how regression testing (at least in my little world) works. No questions please, just enjoy (or not, as the case may be).
Thanks,
Constantine