Errors are among us

A year ago, I didn't know anything at all about quality assurance, because I'd always written code for my own use. But since we studied project management, software quality, and all those things which make that I don't code alone anymore, I realized that errors are far more important than I though.

In fact, the way you deal with errors is the most important. You may know this sentence: the later an error is discovered, the more money it costs. At the early stage of a project, an error costs nearly nothing, because you only designed your project on paper; when a defect is found during development stage, you have to spend - or loose - time to fix it, maybe forcing you to change your design; etc. And the worst case is when your customers find a defect.

Thus, any software project should really consider handling errors, because there will have some, whatever you do. Even if you have the best developers of the world, even if you over-pay them... mistakes will happen, and the way you deal with them can make a huge difference.

The Impulse project, as I eluded earlier, aims at finding a reliable workflow. And you might have already guessed it: we need to handle errors well. So I asked Rémi to set up funny things to handle them easily. As matter of fact, it should not be hard to follow the workflow, because when it's hard, developers do not want to bother with it.

Here are the steps of our process, aiming at finding defects as early as possible.

Unit testing

Each unit of code (in C++ it is often a single class) should have associated tests, and of course, should pass them. Ideally, tests have to focus on a specific point of the specs, all tests should cover all the specs, and they shall be written at the same time than as units themselves...

Well, it is important, but forcing the developers to follow this process is the best way for them to hate unit tests. Developers have to become aware of their benefits, and to know what is really important to test. For instance, the global behavior is worth checking it, but the setter and getter may be simple enough not to test them.

The tests are also the first real application of the unit, before it becomes integrated into the whole project. Therefore developers should enjoy doing them!

For instance, here is the (only) test I wrote to check my BezierFunction class. Self-explicit and funny. I admit that I had no time pressure.

Each developer have to test their modules before continuing in the workflow. At least, they must verify that it compiles, i.e. it will be usable without breaking the development.

To make test development easier, a small library contains functions and macros to test or compare values. There is also automated tests, to launch a set of tests with a single command line.

Team review

This is the big new step: I decided to impose team review for the code. I won't talk for pages about reviews, their benefit and so on, you may found hundreds links on your favorite search engine. I also give two links at the end of this section.

However, why I have imposed team reviews in brief:

• We can fix errors early in the process.
• We know modules, which others wrote: everyone has a better global vision of the project.
  
• The best developers help the not as good ones to improve.

And it turns out well: we use Google Code to manage the development part of the code, and the tool includes a code review tool! So why bother to set up our own? As Rémi stated in his article about the development workflow, provided tools are enough for our needs. This one allows us to put comments on every line of code, thus making discussions on the spotted defects. A rule of thumb is that these discussions may not exceed a few messages: the code review does not aim at finding alternative solutions.

Here is how the whole process should be:

1. Each module has its own repository (an upcoming story about our Mercurial architecture may be coming). During development, developers may push into that repository.
   
2. When a module is ready to be reviewed (and it passes all the unit tests), its code is frozen and a code review is created. The developer in charge of the module assigns another developer as the review moderator (who may or may not work on the module as well).
3. The developer in charge introduces the module and the changes made since last review, if any.
   
4. Every developer who does not contribute to this module read slowly and carefully the code :) They post comments through the review tool, and the moderator avoids that they turn into trolls!
   
5. Once every spotted problem is fixed, the moderator validates the review. The module's development continues in its repository. (No regular development is allowed in the testing repository, except for bugfixes.)

Code must have passed a review before it is allowed to integrate the testing repository. Any review may not move on to a repository push: a code may have several reviews before being integrated. The review is part of the development workflow, therefore all these steps happen in the module repository. Its purpose is to improve code, not to act like a filter for the QA.

So we are going to use this workflow on the project. I didn't say this was the best: the project will show what is good and what must be improved. Since we never did code review before, the results are a total mystery.

If you want to know more about team reviews, I liked in particular this white-paper: http://smartbear.com/docs/BestPracticesForPeerCodeReview.pdf (guess what: this company also sells a tool for code reviews! I don't advertise their tool, I am just saying that this document is easy to read and has cool graphs). If you want deeper information, it may be worth reading these articles: http://www.processimpact.com/pubs.shtml#pr.

Quality Assurance

The QA has its headquarters on the testing repository. Since all the code in that repository passed unit tests and code review, the QA should not deal with deep code problems. Its two main focuses are:

1. Integration issues. Indeed, modules were developed inside their own space, and problems may appear when they are grouped. (This should be seldom since the code architecture is well defined and the module repositories must regularly pull the testing repository.)
2. Game testing: rendering, control behaviors, typos...

When a defect is found, it is either directly fixed when it is small, or an issue report is created. Here again, Google Code provides a tool for issue report management. An issue may be assigned to one (or more) developer(s), its status may change, etc.

When both the QA and the project managers agree to release a version, the testing repository is pushed into the stable repository, and a tag is added. Exceptionally, critical bugfixes are allowed in the stable repository.

We hope that this workflow will allow us to handle defects effectively. It is both easy to understand and apply. The Impulse project should show us what are the good points, and what must be improved. And of course, we will keep you posted about.

Why developers should not have to think

Because the minigame is such a small project, but because we still want to gain experience from it, designing the development processes and workflow of this project has been quite an interesting challenge.

When we started thinking about it, Jonathan threw these contraints at me:

• The code must be peer- or manager-reviewed
  
• We need a separate QA departement, with testers playing with an already reviewed version (no playtests of the development branches)
• Application components must be isolated enough so that we can develop them at different paces and submit versions to be tested at different times, without ending with a half-coded module in QA's hands
• Stable versions must be released on a regular basis, once per week for this project
• Automated unit tests would be really nice
• A build server reporting code breaks and running tests could be really cool too

And because at this time I started to feel a bit uncomfortable about running such a heavy workflow for a one-month project, we discussed it and ended up with the following conclusion: "Ok, we're gonna do all this, but we definitely have to keep it lightweight".

And keeping it lightweight is not only about style... If we fail that part, then we put the project in danger for two main reasons:

1. Obviously, if the processes are too heavy, people will lose too much time applying them and the project will quickly get very late.
2. If the workflow is too complex, everybody will eventually get lost in it, work around it, thus making it completely inefficient.

Moreover, both of these problems would probably impact moral and discourage the team, which is one of the worse things that could happen to us :p

Don't-think-about-it philosophy

In order to find an appropriate solution to our problem, we can start by defining a general working philosophy. Because we need the team to realize a huge set of things in a short amount of time by following relatively hard constraints, the more burden we can remove, the better. Part of removing the burden is obtained by applying what I call the "dont-think-about-it" philosophy.

That's why you can buy a golden fish without bringing water with you to the store.

Really.

Because it's annoying (and risky for the car) to carry a water bag, because you wouldn't know what kind of water (salted?) to bring, because maybe you didn't even plan to buy a fish at this moment, etc.

But while marketing professionals seems to have figured that out for a very long time, computer science looks a bit outdated on this.

The point is that when you have to think about a lot of things at the same time, the chances that you get something wrong increases very, very quickly. In a similar spirit, when you have to check many things before doing something, it is likely that you will forget one or two and then realize that this was wrong.

Thus, when designing a workflow, we should strive towards two main goals. First, always reduce the likeliness of something wrong done, by reducing choices and checks a developer has to do in everyday operations. And second, we should always design the system by taking into account that mistakes and errors will happen.

If this sounds familiar, it is normal. That can almost entirely be found in any good UI design course. The fact is that things doesn't have to become overly complicated when we get to developer side.

In our project, this state of mind translates to as much sentences as possible looking that way:

• Developers should not be scared of pushing/committing something on the central server. Never.
• Finding which version of the game a bug is related to should not require any research in the repositori(es). When a bug pops out, everybody should know on which version it has been found.
• Nothing in the compilation process should have to be handled manually.
• No special action (to be remembered) should be taken at each commit/push, including triggering the build server for a new build.
• There should not be any repetitive and error-prone work (e.g copy/paste/replace). Obviously this kind of thing can be hidden in scripts or small IDE tools, but it's generally much better not to have it altogether where possible.
• etc.

The tool side: trying bottom-up

Because whatever solution we choose we will have to implement it on existing tools, here is a small review of what tools we need, and what alternatives we considered.

The code must live on one or more centralized repositories. It might seem obvious at first sight, but DVCS like git or mercurial allow developers to push/pull directly from someone else's repository, which would allow server-less source control. But in our case, the team is exploded between numerous European countries, and we cannot expect a developer machine to have the same disponibility level as in a company local network.

Thus we must make a choice between the hosting possibilities:

• Managing our own server. This is the most flexible option, but it will inevitably take a lot of time to set up and administer everything.
• Using a free online service, such as Google Code, GitHub or BitBucket, and add missing features (such as a build server) on our side.

For many reasons, we chose the latter solution. These reasons include administration burden, server reliability and backups, and more generally the fact that the free online offers are enough for what we need. We chose google code because most of the team is familiar with it and with other google tools (e.g docs), and because we wanted to give Mercurial a try on this project, to set up a real DVCS workflow (see below).

To store the different game versions (such as development, testing, releases...), we need a powerful version control system, supporting a lot of branching and merging. This led naturally to Mercurial on Google Code, which allows multiple repositories for free. The repository architecture is made with the following goals in mind:

• Stable releases should be separated on their own repository
• QA testing and bug fixes are done on their own repository
• Feature development is done in several repositories, one per architecture component
• Features can be integrated separately into the testing repository
• Repositories have to communicate. For instance, bug fixes done in the QA repositories must be integrated in the development branches. Symmetrically, reviewed components must be pushed to the QA for testing.

While this could have been achieved with SVN branches, Mercurial seems to have a much more lightweight and clean way of doing it. And most importantly, we wanted to try it :p

Choosing Google Code as a back-end service also helps a lot with the integration of other services we need. Code review can be done directly in the code, on a line-by-line basis, and issue tracking is tightly integrated in the web UI and with commit messages. This is a great advantage because it means developers don't have to go elsewhere (i.e another service, another URL) to do these things. The wiki is equally integrated, and will allow easy project documentation.

As the game is cross-platform and we don't want to lock the developers in only one-tool (i.e a specific IDE), the build system is based on the powerful SCons, which is easily callable from any IDE project, and allows for great build flexibility and cross-platformness. As the build is a complicated topic, another article will be written to explain how our build system was thought and implemented over SCons. The main design points are the following ones:

• Cross-platform and cross-compiler. Users may chooses to reduce compiler choices, but the toolchain must support all of them, theoretically.
• Component concept. Components are something between an IDE project and a package-config file, generalized.
• Explicit component dependencies. This will allow for component dependency graph generation, avoid duplicated configurations and kill dependency cycles.
• Powerful configuration. Everything is Python.

We also make use of a build server which is automatically triggered by Google Code, through Post Commit WebHooks. This allows us to be almost instantly notified if we break the build or some unit tests.

Finally, unit testing is kept simple and flexible. That means we allow both automated and non-automated tests to be written, and that we developed a very simple library to help with test development. Actually, that library is just a header with three macros in it, such as CHECK(expression) which will print either a positive or negative result to the output, along with information about file and line where errors happened.

Allowing non-automated tests is also an important point: we don't want developers to waste time trying to fit a simple module test within a complicated yes/no framework, especially when this test is easy to verify but hard to automate (is that texture yellow?).

Everyday life made simple

As stated before, the ultimate goal of this tool combination is to make developer life simpler. Maybe it is not ideal, but I believe we have reached a fair degree of simplification. The build system handles weird compilation details, the build server handles running the tests as often as possible, writing tests is easy, pushing is not scary because the repositories are separated, bugfixes can be instantly imported in development branches...

But obviously, as this workflow is new, we cannot (yet) know what will really work and what will not. More about that after the project has finished and remarks have been collected...

Challenge: create a mini-game in one month!

Vacation is over! And to get back to work, we are setting ourselves a major challenge: creating a mini-game in a single month.

Well, we don't want to make a blockbuster in only one month. We are aware that it will be small: a few levels, with simple gameplay, limited artwork. Sure, we hope that it will be fun to play, but this project have several other interests.

The first is of course making a game in itself :) Indeed, almost all of us never made a professional looking game, in a well-organized team. This project aims to follow the complete steps of creating a game, from concept to release. It will be instructive for everyone: it gives us the opportunity to understand the game creation process, and to complete our (always starved) knowledge.

Another main interest is to allow us to experiment new methods and workflows. The project lasts only one month: perfect to see whether they are good and have to be applied to our other projects. Among other things, when developers finish a module, it is first team-reviewed and then integrated into the main development branch, only if validated. We also test new tools: code will be versioned under Mercurial (Hg), a private forum will allow more effective communication... this blog is also part of the renewal: we will keep you informed of our last advancements and technology choices!

And to complicate even more, we only have a single month to do it!

The project runs from September 16th to October 14th. Every few days, we will post news, along with technical articles, so subscribe to the RSS to stay informed!