1. INTRODUCTION
This document serves as the first training guide for Game Testing. The document explains the fundamentals of game testing at a level appropriate to a new tester or someone who is requested to help out game testing for the very first time. The reader will learn the various aspects, background, and involvement of testing for game production. Many of the issues and areas of interest presented in this document will be covered in more detail in accompanying documents. These accompanying documents will provide a higher level of sophistication, detail, or depth, which is required, as testers become more experienced and knowledgeable.
The information in this document is presented in a layered approach (known as top down) so testers can develop skills through on-the-job training and gradual learning. Other documents will cover the testing methodology, the testing processes, strategy and approach, and the development of a Test Plan. Amongst all this documentation, this document should be the most enjoyable.
Any questions about the information in this document, or about testing in general, should be directed to a Test Lead or the Software Quality Director.
2. GAME TESTING, OR GAME PLAYING?
“That’s game testing, right?”
“They pay you to do that, so that is what you do all day.”
“You play the game until it breaks.”
There are many game development companies that strongly rely on playing to test a new game. We must make a clear distinction between playing and testing a game. The emphasis of Game Testing is a systematic approach that looks at all areas of a game to ensure that the software is bug-free as well to confirm that everything works as required. This is not to say that game playing is bad, playing a game is still the most effective means to test the interaction of a game and the game components. However, it is strongly believed that game playing alone is NOT sufficient, and we need diversity to ensure proper test coverage of a game.
A computer game is enormously more complex than a typical software application. Most games are made up of a group of essential components, which give the user visibility, playability, and interactivity. Examples of these game components are:
§ the play engine that allows the game or the user to direct the play and gives the resulting commands to other components
§ a rendering engine for the 3D or 2D graphic for the character, the world, and the texture including colouring
§ Artificial Intelligence (AI) for the game logic and/or the opponents
§ database that is used by the game for AI, statistics, randomization, etc.
§ animation
§ the functionality of the objects, world, models and characters including all the moves
§ the game control device (i.e., the use of controller, keyboard, mouse, etc.)
§ the various front-end screens like selection menu, pause menu, help menu, etc.
§ sound, special effects, camera views, networking play, platform dependent service layer and objects, and so on.
To work with a PC, you don’t need to know the “nuts and bolts” of the PC architecture or the Operating System in order for you to use it. However, understanding the function and interaction or integration between each of these components is important to a game tester. This knowledge helps a tester determine how to test a game. A Software Game Tester is expected to get familiar with the game architecture as well as the game components. However, this component structure should not overwhelmingly dictate what you should test for any particular game.
The other reason testers need to know about these game components is for bug investigation. Often times when testers encounter a bug, they only see the “symptom”, the root-cause of the problem could be a bug that exists in the associated component layer.
3. GAME TESTING? OR APPLICATION TESTING?
For most typical applications, such as, a client/server application, an Internet Browser, or a database system, the software is designed to give the user easy access to the function. The underlying logic and application function must be predictable. The workflow must be well structured. The behaviour must be clearly defined. The interaction for the process must be efficient in terms of time and the number of steps to get something done or to obtain the desired result.
Fundamentally, business applications are transaction based. Computer games are mission based. For the most parts, computer games are hiding the (game) functions from the user because this is what the user expects while playing a computer game. The user must search and explore the game as much as possible.
§ Predictability is the exact opposite. The game wants to preserve the mystery of the world, the quest or the fantasy.
§ Efficiency is the exact opposite. The gamer must explore and discover what the game does, the harder it is, the more time it takes for the gamer to obtain the “desired result”.
§ The user interface and the flow are intentionally limiting. With the “randomizer” or “Artificial Intelligence”, most computer games change the way it interacts, as well as the way it looks for different circumstances or game states. Some games even try to outwit you from getting to know the flow and what it is going to happen next.
§ Adding to that, there are enormous amounts of changing details, each slight difference appeared on the game could affect the game’s behaviour to a large extent (e.g. an object trigger on the screen may lead to an explosion or open up a secret passage).
To illustrate how to use a systematic approach for game testing, you will break down each game feature into its smallest components. For examples, the world or course (the track, the terrain and so on), the textures (the graphics of the world), event triggers (the environmental object) and their interaction (e.g. the sound and the action). Using these examples, you would expect to move the character down the track and then hit at the exact place that the event trigger would click in. There could be bugs that you can find like clipping (polygons are rejected or cut when touching the edge of, or lying entirely off the screen), drop-out (polygons disappeared when getting closed to the camera, incorrect collision detection (objects are allowed to pass through each other), grayed out (when an event trigger cannot be selected or initiated) and so on.
4. GAMES ARE MEANT TO BE FUN
As equally important, we also test the intangible attributes of a game, and ask the questions “is the game fun, is it exciting?” and “does it look realistic?” Fun and excitement is subjective, and there is no fixed rule to assess a particular game play whether the actual implementation is right or wrong. Because of this, it makes some people think game testing is almost an art form.
A human character is often made to look like a human (walk, move, punch, dribble a basketball, etc.) sometime certain moves (e.g. a roundhouse kick, or a slam junk) are exaggerated for better visual effect. In the myth of testing the intangible attributes of a game, we must understand there are a number of factors that contribute to making a fun game, and there are so many things in the game that can be changed to make the game more fun. Learned from our past experience, a new Game Tester should be aware of the following:
4.1 Testing for Fun and Excitement
§ There could be comparative and similar game titles available in the market, you can make reference to those games as “what does the competitor do better?” or “which of their features must be built or improved upon in our game product”. This helps us to have an “out-of-the-box” thinking to look at our own games.
§ You must have a “gamer” mentality and test it “for the love of the game”. You don’t necessarily have to be a hardcore gamer, but be sure you like video games.
§ It is important to have a good understanding of the theme and the differences between various video games (e.g. role playing game, fantasy game, space simulation game, strategy game, sport game, etc.)
§ A video game needs to be challenging. The harder it is, the more fun for the gamer to get to the “desired result”, such as, completing a mission, or conquering the quest. However, everyone wants to be able to win eventually. Therefore the game must have progressive levels of difficulty, and it should encourage learning and developing skills through actual game play.
§ How does the scoring system work? Is the scoring fair, are the “high scores” set too high or too low?
4.2 Testing for Realistic Game Play
§ Games often follow factual and historical rules. We do not need to be totally accurate with every single detail, but we must have a good understanding of the “game story” and its background information. For examples, a sports game must have certain rules or ways to play the game, there are game and player statistics, and there could be certain moves that make the game interesting. For games that have a historical background, the artwork and objects should be consistent with respect to the “past-time”.
§ Simulation is very critical to make a game look and feel “real” together with good animation (like Jackie Chan’s unique fighting style). When Artificial Intelligence is incorporated in the gameplay, it is used to simulate defensive and offensive play of the opponent characters (e.g. like the opponent team/players in a sport game, or a computer controlled enemy fighting against you in an action game). AI should make the opponent look “realistic”, and all of your opponent player(s) must be able to perform at a level that is comparable to what and where you are.
§ Animation must look realistic and cool, and the animation sequence should be rendered at a reasonable frame rate (the target frame rate is 30-60 frames per second depended upon the platform). Often a good animation sequence may suffer from a poor frame rate after optimization, and the tester should identify any unnatural and awkward movement.
§ There are predefined attributes of certain objects and features of the game that must be tested for “reality”. Some of these examples are, the strength of your enemy, the way your enemy engages into the fight and the fight scene, the amount of damage for an explosive object, the effect of your regular moves and special moves (for example, the special move should have a better visual and/or sound effects than the regular move), and so on.
5. SOFTWARE TESTING
Software Testing verifies that the interfaces and integrity of the software component function (in some cases, including the tools that are used for the game development) and demonstrates that the components meet their game design and technical requirement.
The Software Tester has a coding background, so is expected to look into the game code. The Software Tester uses a run-time environment (e.g. Ptui, an in-house developed debugging tool for the Playstation game, or some other tool for a different platform), feeds the code or chunk of code with input (data, setting variables, etc.), and then analyzes the test result:
§ if the test runs successfully; without any major fault or failure,
§ if the expected result is achieved,
§ what are the run statistics – information is passed to the developer for review,
§ if the run result is consistent with the game statistics (e.g. in terms of scoring, number of actionable moves and/or animation sequences used during the actual game play).
No one can test a program COMPLETELY, i.e. testing every single path of the logic and all the possible variations of input, interfaces, data generation/manipulation, and then output. Our software testing strategy is to develop good, sufficient and effective testing.
5.1 Testing Technique
The process and techniques that are commonly used for software testing will also be applied for Software Game Testing, these are some of the techniques that we have used:
§ Code Inspection
§ Incremental Focus Testing
§ Data Analysis
§ Path and Flow testing
§ Algorithm Specific testing
§ Artificial Intelligence Analysis
5.2 Test Automation
Some people argued it was far too complex and variable to build any kind of a test suite for games. Automation could only be useful in checking if and how many of the objects and the animation actions got used in the game, but most of them were no help for the actual game testing. We DO have a different argument on this Test Automation issue. We believe it would be extremely useful if we have a good understanding of the game (from the Game Tester’s view) and a good understanding of the game technology (from the Software Tester’s view).
Test Automation does not come in light automatically. We must analyze and formulate the “test automation” requirements and work with the game developers. There are many practical opportunities that we can take advantage of automation or automated game play, for example, play a complete full season of a sport game, speed up the game clock, get a troop of army to conquer the whole world, and so on. Once we have determined the test automation requirements, the game developer will help to implant the “automation enabled” code into the game. It is important to note that sometime this test automation code may or may not be required to be removed from the Final build of the game.
6. FOCUS GROUP TESTING
Radical uses Focus Group testing to receive feedback from targeted audience about the quality of a game. The game team uses the feedback at many levels of generality, and even functionality of the game to look for improvement ideas. The focus group review is also used for market research and comparative product evaluation.
For a new game title, the Focus Group participants help to generate and confirm design ideas by focus testing similar or comparable game titles that are available in the market. As the name implies, the Focus Group testing is to look into some specific areas of the game, e.g. what are the popular and likeable features for a new game. For a pre-release game, the Focus Group testing is to receive feedback from the targeted audiences, e.g. what features are interesting and fun and what not. The results is used to identify any tuning and critical changes to be considered for the game, and at the same time to have a sensing of its popularity before it arrives at retail.
On average, we conduct 3-6 (the average is 4) Focus Group sessions for each game. The result of this Focus Group study is very useful beyond the current game production. When we work on a game that follows the sequel of a previous game title, the feedback and suggestions are often used to formulate the next game’s feature requirements.
7. DEVELOPMENT CYCLE AND MILESTONE STAGES
The entire software development process for a video game is made up of “pre-production” and “production”, or a pre-production cycle and a production cycle. Pre-production is to design and build various tools and libraries that allow designers and artists to iterate on their work, and to implement global and reusable components delivered to the game team. The requirements of testing will be focused on Software Testing if needed.
For the production cycle, the Producer and the Project Manager manage all the required work under a project. Project milestone dates are set up so we can monitor, measure and report the progress. Each milestone date will be clearly defined in the project timeline with a set of milestone deliverables. This is a contractual requirement that we must meet our milestone in terms of deadline and deliverables. The number and definition of milestones vary from project to project, but they all have a high degree of commonality. We can generalize certain milestone dates that are important to QA:
A) Game Requirements: The Game Requirements phase is often considered NOT part of the Game Production cycle because the requirements are developed and used as part of the contact for the game publisher. Bridging between the Game Requirements to the Technical Design, there are intermediate deliverables produced by the Game Team like the Roadmap and Game Design Document. Please note this information is important to define our baseline testing requirements.
B) Technical Design: The Technical Design Specification together with the Feature Specifications are used to develop the required testing documentation like Test Plan and Testing Requirements.
C) First playable build: This is the milestone where we can have the first “look” of the game. Very likely the first build of the game does not have a game flow yet, but we can look at pieces of it using debugging software. We expect to have multiple iterations of the build available throughout the production cycle with incremental improvement of the game’s functionality (e.g. more features are added/incorporated into the game).
D) Alpha: By definition, reaching the Alpha milestone means that the game clearly demonstrates the game play, i.e. all the critical features, the style and the art work including sound. From the software development’s perspective, all the game functions are coded, and there are no significant coding risks left.
E) Beta: The Beta milestone means all game features are complete and tested, also some other desirable but non-critical features have been added to the game. The focus of beta is on finishing up. The game is to be tested if it is customer-ready and market-ready. From the software development’s perspective, all functions including additional features are coded, and there are no significant defects left outstanding.
F) Final: At final, the code is frozen and no more changes can be made to the code for any reason. All the cheat codes will be removed from the game. The game is ready and configured as a commercial ready version. Together a list of what files and game components will be packaged onto the CD for delivery to the publisher using the standard media.
8. QUALITY ASSURANCE CONSIDERATIONS FOR A PROJECT
Our current resource strategy is a full-time Game Tester and a part-time Software Tester for each project. Their actual assignment and the start and end dates varies from project to project. As a rule of thumb, the Game Tester and the Software Tester would start roughly at about the same time, and their first assignment is to develop all the required testing documentation at about 50% capacity. The Game Tester gradually works up to 100% capacity to Final after the first playable version of the game becomes available. The Software Tester is working 100% during game production to the first 2-4 weeks of Alpha. By that time, all the game code should be pretty much completed, and we don’t expect any significant changes required for the game code. During the game production, the Game Tester is also responsible or takes part in Focus Group testing.
There are many reasons that drive and affect the decision making process for a project. These reasons are beyond the scope and control of QA. QA must realize and support the decisions made by the Producer who strives to complete the project on time and within budget. Some of these challenges are:
§ tight and difficult production schedule
§ insufficient testing resource
§ resource changes
§ project slippage
§ scope changes, such as, late design changes and additions.
QA acts in a support capacity and works with the team so they know how much testing time is required, how much it can be compressed and at what point and to what extent the amount of testing may be comprised. However, our mandate does not change and we must thoroughly test a product, and test each build of the game in several iterations. There is a minimum amount of time and effort required for the testing activities, and the game team must be aware of this threshold (for example, at least 4 week elapse time between the milestone dates of Alpha, Beta and Final). In order to deal with tight and difficult timelines and requests, the emphasis of testing is more focused on testing the tangible elements of the game rather than the intangible elements of the game (i.e. make sure the game is playable, not so much on how playable the game is).
9. THE TESTING PROCESS
Game production goes through a number of stages. Most of the stages are often described sequentially but the underlying tasks are often proceeding in parallel, especially during game production. Testing and fixing can happen at any stages in the project cycle. However, the cost of fixing an error increases exponentially as the project progresses.
To develop a high quality test plan and testing strategy for a game project, we use a “Full Cycle Testing” approach. The objective is to identify and fix the error as early as possible so a quality product will be delivered at the end. The focus for Full Cycle Testing is on 1) validation for accuracy and correctness, and 2) verification for completeness. The results of the Full Cycle Testing from the early stages of the game production will be used in the test design, and subsequently the creation of testing deliverables like test plan, test script and scenario testing requirements.
As a brief outline of the testing methodology, here are some of the things that the Game Tester and the Software Tester would do in the early stages of the project. They are expected to be working on it together before they get to work on their own portion of Test Plan:
§ Read the Requirements and related documents to come up with the testing criteria and completion criteria for the game, the questions you would ask include, “Are the requirements right?”, “Are they achievable?”, “What are the measurable and quantifiable targets?” and so on.
§ When you work on the Roadmap and Game Design documents, you check for “Is the design complete for the Game Requirements?”, “Are they testable?”, “Is the design realistic and doable?” and so on. You come up with a list of testing requirements and a high-level quality and test plan.
§ When you work on the Technical Design and Feature Specification, you review the documents to check for “Is the technical design complete with all the required features?”, “Is the design reasonable and realistic?”, “Is the game architecture complete for what is needed in terms of tools and libraries?” and so on. Based on the review, the test plan, scenario testing requirements and test script are produced.
For game testing, the test plan should cover all the “positive testing”, i.e. what/how the game can be played, and the scenario testing will address the “negative testing”, i.e. special conditions and circumstances that the game should be able to respond correctly.
For reference purposes, you may read the books on Game Architecture and Design, and Testing Computer Software.
10. DEFECT REPORTING
Defect reporting is one of the most important communication tools used in Game Testing. The defect (or known to you as “bug”) is documented in a defect tracking software (currently we use PVCS Tracker software) to convey the problem information found by the tester to the programmer or designer of the team. It is important that the bug should contain all the essential information so that the problem can be clearly understood and then acted upon by the appropriate person.
From our experience, if we started this defect reporting process too earlier, say, at the first few playable builds of the game, we often found far too many bugs that were already known to the developers because the game was understandably “buggy” and unstable. At the end we had far too many meaningless bugs that were left unattended in the defect database. In the early stages of the game production we encourage the testers to discuss their problem findings with the developers directly. We also encourage if the developer agrees to enter the known problem into the defect database. For legibility of bug reporting, it will start when a stable and playable game build is in place, roughly 2 weeks before we hit the Alpha milestone date. When doing so, we must have the “go-ahead” by the Producer and then notify the entire game team.
The defect information is solely for the purpose of reporting bugs. We often use the defect information to generate useful metrics and benchmarking statistics (e.g. what game areas have the most problem, how soon we fixed the problem after it was identified, etc.) in post-mortem. We never use this defect statistic to measure someone’s job performance (e.g. the person whose code has the most problems) nor to evaluate how productive a Game Tester is (e.g. who reports the most bugs).
11. TIPS AND HINTS FOR TESTERS
Here is the description of some of the important responsibilities and duties as well as tips that you may find them helpful:
§ Report a defect summary every Monday once we hit the Alpha milestone date.
§ Do a defect summary review after the game production is complete.
§ Get educated and be knowledgeable in some of the newer hardware and game technology.
§ Participate in some of the design and/or review meetings by asking the Technical Director of the game team.
§ The start of a project is the ideal time to read the system documentation and create testing plan. A good test plan should not be inadvertently affected by any fine-tuning of the design document.
§ Know the rules of the game that you are testing.
§ Pay attention to some of the trivial details because they are NOT trivial to the publisher like checking text in the menus, the naming standard and requirements for the platform manufacture like SONY or Microsoft, and the copyright information.
§ Provide your Technical Director with a weekly status report, highlighting what you have accomplished this week, what you plan to do in the following week, and any major issues, problems and/or concerns.
§ Ask for HELP if needed. This is the culture of our company, and we are all here! Don’t send a lengthy email to someone who is sitting closed to you.