1. INTRODUCTION
Effective testing comes from a well-structured approach and a well-defined testing methodology so the game product is highly satisfying to our Publisher and the game player. In contrast, poor testing results in a buggy game or software that gives rise to a long stream of repeated testing and project delays.
The purpose of this document is to explain the testing methodology for game development. This document is intended for the person performing the testing. The topics and the level of details provide an interesting and useful set of information for the Game Tester or Software Tester. There are a number of Quality Assurance and testing initiatives that are covered in this document, including a testing strategy, testing processes and techniques, test coverage criteria, Testing Requirements, a Test Plan definition, testing documentation, and “Full Cycle Testing”.
There are few, if any “fixed rules” for this testing methodology; however, there are many suggestions, ideas, and guidelines for improving the quality and effectiveness of testing for our game projects. Hopefully, testers will be able to learn, understand, plan and carry out effective and efficient testing in a structured manner. If you have any questions about the information, please contact your Software Quality Director.
This document must be used in conjunction with the other companion QA documents
1. Game Testing Primer,
2. Testing Design for a game project, and
3. Defect Reporting Guideline.
2. TESTING STRATEGY
2.1 SCOPE AND DEFINITION
In a simplistic view, testing is to identify bugs found in the software, so the problem can be removed. There are different forms of tests and testing that can be categorized as “Black-Box” testing and “Clear-Box” testing (“Clear Box” testing is also known as “White-Box” testing in the software industry). Their testing objective and overall processes are indifference (e.g., test planning, test design, testing execution, regression testing and bug reporting), but their focus of attention puts emphasis on different aspects of the game:
§ “Black Box” focuses on the functional or the playability aspects of the game. For examples, testing the user interface (e.g., the selection menus and the use of buttons), the “look and feel” (e.g., the graphics and animation), and the actual gameplay.
§ “Clear Box” is focus on the architecture and integration aspects of the game software. For examples, the use of a database, pipelines, the interaction/integration of game components like the rendering engine, the AI engine, sound, and so on.
For Black Box testing, the tester must know how to play the game (e.g., use of the game pad, know the rules and the game flow). For Clear Box testing, the tester must understand what coding is. The Software Tester uses a run-time debugging environment, feeds the code or chunks of code with input (i.e., data, setting variables, etc.) and analyzes the test result.
2.2 TESTING PHILOSOPHY
Testing is NOT a single person’s job, nor solely the responsibility of the Game Tester and the Software Tester for a game project. Every team member working in a game project must have “quality” in mind, and each person is responsible for the accuracy and completeness of the work that he/she produces.
This testing methodology is NOT the only process and it should NOT be used in isolation. The reader must be aware that this testing methodology is considered as an integral part to the Game Pre-production and Production processes.
In reality, no one can test a program COMPLETELY, i.e., testing every single part of the game using different and all available characters, so triggering different path of the logic and all the possible variations of input, interfaces, AI, and then output. Our testing strategy is to develop excellent, full-coverage, and effective testing (i.e., 80/20 rule).
3. TESTING PLANNING AND TESTING REQUIREMENTS
3.1 IDENTIFY THE TESTING REQUIREMENTS
A requirement is an objective that must be met. The tester must understand most of the game requirements and translate them into functional terms (e.g., what needs to be tested, what things are testable and what not, what are the targets and measures, etc.), leaving the design and implementation details to the developers.
As part of the testing requirement development process, the testers read the system documentation, gather all the visual and written information, analyze what and how the game components can be tested. It is the responsibility of testers to read all the relevant documentation so they can share (to understand and appreciate) the mission of the project. Please refer to the “Full Cycle Testing” section about how to conduct a quality review.
You are required to develop a Testing Requirements document for each game by outlining what and how the game and game components will be tested. The document includes:
§ a list of features,
§ the specifics of the internal design and external designs of the game. This may require a description of the possible implementations if it makes the testing requirements easier to understand (e.g., certain theme of the game, the characters, the animation, the AI, cinematic or camera view, and so on). For example, to test the multi-directional fight action for the Chan PS2 game, you must make reference to the use of “Ring of Engagement”, describe how the opponents engage into the fight scene, and what you expect the single/combination fighting actions.
§ a testing structure detailing if and how Game Testing and/or Software Testing is applied (i.e., in a spreadsheet format for the items identified above),
§ the testing criteria (e.g., ideas for testing), and
§ the completion criteria (e.g., what are the expected results, what does “something is done” or “something is working” mean to you in game testing?)
After the testing requirements are identified, the Technical Director for the game and the Software Quality Director will review the document to confirm scope and priority. Following the test requirements, the testers work on their own test design and develop a Test Plan and Test Cases. Any testing dependency requirements must be identified and communicated to the game team so the game code is “test-enabled”, i.e., what kind of cheat codes, or “test automation-enabled” code are required?
The testing documentation is expected to be developed in the early stage of the project, i.e., draft testing documentation is produced when we have the first playable build. It is important to note that the testing requirements will not cover every single detail of the game, but it must cover testing all the contractually required elements (e.g. specific features and the major game functionality). You can obtain this information from the game Technical Director.
3.2 DEFINE THE TIMELINE REQUIREMENT
When a game project is pressed for time, we must recognize the existence of a threshold point where sufficient timeline must be provided for the testers to perform:
§ a number of iterations for testing each new or updated game features,
§ a complete cycle of regression testing for each build,
§ sufficient regression testing of the previously Critical, Closed bugs, and
§ a full regression testing expecting to test every event/world/environment object and triggers in the game for Alpha, Beta and Final.
This “threshold” point varies from game to game, the tester is expected to communicate the “bare bone” testing requirement with the Producer and/or Project Manager.
4. GAME TESTING AND TESTING TECHNIQUES
The Game Testing Primer document describes a few aspects and focuses of game testing. This section outlines a systematic testing approach, and explains how you break down a game feature into its smallest testable components:
4.1 GAME COMPONENTS AND BREAKDOWN STRUCTURE
Systematic game testing means examining every part of a game. These parts include:
§ the menu and the menu functions,
§ art (character model, texture, terrain or world, crowd, objects, etc.),
§ animation (the like and feel of the movement, realism, frame rate),
§ sound and the sound effect (in conjunction with the facial animation, e.g. lip sync, and the animation sequence),
§ music,
§ camera (cinematic view, zoom in and out, replay),
§ game flow and logic,
§ world/level/scene,
§ the player attributes,
§ the action attributes,
§ the condition to advance to the next level (what are the rules?)
§ the use of environmental objects,
§ the event/object triggers, and
§ the scoring
§ progressive levels of difficulty,
§ the AI logic (for both defensive play and offensive play; player movement and positioning),
§ statistics (pre-game and in-game like player statistics and high score),
§ title screens,
§ NIS (Non-Interactive Sequence),
§ SFX (Special effect)
§ any movie clip,
§ the game pad,
§ the use of multi-button actions (also called button mashing),
§ the ease of use of the button functions,
§ the shock/vibration effect of the game pad,
§ the use of digital and analog mode
§ legal text, and
§ the game options (game start/menu selection, hints, game pause, pause menu options , and scrolling, i.e. cycling through the available options on the screen, etc.)
4.2 GAME TESTING TECHNIQUES AND “TIPS-N-HINTS”
Here are the specific details and the testing techniques:
§ Must examine the entire screen and not just a small part of it.
§ Familiar with the game rules and test the gameplay against these rules.
§ Test for clipping (two or more polygons or polygon objects overlapping each other or canceling each other out).
§ Examine the overlap (where a semi-transparent object shown on top of another solid colour object and/or the background), and check if the overlap is appropriate in terms of size, placement, the purpose and the information that is provided).
§ Test for incorrect and inappropriate collision (the condition when two objects should be colliding). Imagine when two cars collide, they should get bumped by each other with about the same effect (e.g. damage).
In terms of physics, when a collision involves two objects with the same mass, they should generate the same amount of momentum in the opposite direction. In contrast, when a plane hits the ground, should only the plane show the effect of collision because the mass of a plane is trivial compared to the mass of the earth.
§ Move the character through all the available objects and all the levels and closely examine the effect (it could be collision, an event trigger, or an object trigger).
§ Test for grayed out or grey out (when an option or icon cannot be selected)
§ Test loading/saving from a game save device (e.g. hard drive or a memory card) and ensure the correct messages are displayed on the screen.
§ Ensure a “game load” or “data load” message and an appropriate “loading” indication (e.g. loading meter) displayed on the Loading Screen.
§ Ensure the loading time is acceptable (no one likes to wait for more than 20 seconds).
§ Look for micro-pause where slight pauses occur in the game but do not actually cause a full freeze or crash. When it happens, it will temporarily impede the gameplay.
§ Test for the multi-player mode (the game runs on one machine, not on-line gaming) where two or more players engage in one game.
§ Test the game for memory leak or memory overload by leaving the game on/running for a few days.
§ Test end of bound (also known as end of the world or edge of the world). The game should not allow the character to move “out of bound”, and/or the game becomes erratic (e.g. game crash/freeze, the texture becomes deteriorated, the character gets trapped in certain area of the map).
§ Test for platform compatibility. Platform testing requires some extra effort for the PC game and on-line gaming:
§ For a PC game, test for all the supported operating systems like Microsoft Windows 95/97/98/2000 and NT (this includes Install, Uninstall, and actual gameplay). Also, test for different sound/graphic cards and peripherals that are available in the market (discuss with your Technical Director for the make/model of the peripherals).
§ Test the networking for game play over modem(s) at different speed, and Internet gaming (The depth and coverage for Network Testing will expand if the game itself relies on some Sever applications for the actual game play; e.g. multi-player mode).
§ If the game is to be published in European countries, test for PAL conversion (currently the game we produce is mostly targeted for Northern America, and we use NTSC).
§ Test for localization (foreign language requirements) if required.
This is a list of common testing techniques used by a Game Tester. This is not meant to be a complete list, an experienced Game Tester will tell you that you should use your mind – this is where a little creativity, a little imagination, and a little bit of “common sense” become very useful.
4.3 SPECIAL CONSIDERATIONS FOR GAME TESTING
Before I let you go, I must warn you about Crack bugs and Placeholder, they are often mistaken as real bugs:
§ Crack bugs (i.e. a bug which in fact is NOT a bug, but someone’s crazy imagination).
§ Placeholder (i.e. something “dummy-up” on the screen, it is used to fill a spot until the correct video, artwork, sound or object is implemented and incorporated into the game).
5. TEST PLAN DEVELOPMENT FOR GAME TESTING
There are two types of test cases; one is testing the functionality of the product; commonly known in the software industry as Positive Testing. The other one is testing the durability and the level of tolerance resistance of the game; known as Negative Testing (it is also known as Stress Testing or Load Testing). “Negative” test cases are created with “breaking the game” in mind, so we will test for certain odd situations and determine if the game knows how to respond, and reacts correctly. Examples of this odd situation are,
§ Load a game without a MEMORY card, or pulling out the MEMORY card during game loading.
§ Running a game for more than 48 hours in order to test for memory allocation and management. When the game component does not handle the use of memory properly, some of the memory may become unusable (the memory does not get recycled). When it happens, a game will become non-playable (e.g. freeze) after a long-playing time.
§ For the playability testing, say, a scenario is developed for the “Head-to-Head” mode in Snowboarding, we have Player-1 boarding very fast, and finishing much earlier than Player-2, which is boarding extra slow. We expect the game should be able handle this extreme situation (e.g. no freeze, the scoring continues to work the way it should be) till Player-2 is finished.
Define Smoke Testing to test a new CD burn. The name of Smoke Testing comes from the engineering lab in testing new car engines. Before a new engine is put out for a road test, the engineers would simply start the engine and see if it works. If smoke comes out from the engine, they would know immediately that some parts do not work properly. Just like creating a new build, we simply run the game, if the game keeps on crashing, we would know some parts are not right. The Smoke Test for a new CD burn needs to be defined. The test plan is a simple list of what main game features and options need to be tested so you are able to confirm a new build is successful. Just like testing a new car engine in the lab, you would start it and let it run for a few cycles.
6. SOFTWARE TESTING AND TESTING TECHNIQUES
6.1 SOFTWARE TESTING PROCESSES
The Game Testing Primer document outlines the role of software testing. The processes that are commonly used for Software Testing are:
A) Code Inspection: review the source code and focus on logic structure, conformity with standards, the use of comments, and the completeness of program documentation and/or definition.
B) Incremental Focus Testing: test chunk or pieces of code by feeding values/input to an isolated module, and analyze the intermediate result from the module. Usually a run-time facility, a debugging tool, a “dummy” front-end, which allows invocation of objects by their underlying method(s).
C) Data Analysis: examine the data in the database; or analyze which part of the system uses or modifies any item of data. By tracing data items through the game play, the tester can validate if data usage, interpretation and manipulation by the modules or objects are appropriate.
D) Path and Flow testing: verify if the correct sequence of objects or components is executed for an end-to-end path.
E) Algorithm-Specific testing: test a specific game scenario or feature by setting (or tweaking) data variables, data values to the code, and execute it in the run-time environment. Different from the Incremental Focus Testing, the testing is to verify a suite of codes that make up a specific game feature.
F) Artificial Intelligence Analysis: generate the run statistic of the programmable moves and plays of the AI components. The result is to validate if all the pre-programmed moves (e.g. side grip on the snowboard) and plays (e.g. Jackie Chan’s combination punch/kick in a multi-directional action) are used. Verify if the AI logic is appropriate and reasonable (e.g. in a basketball game, it would not be reasonable to have the scoring 90% by dunk; and 10% by jump shot).
6.2 SOFTWARE TESTING TECHNIQUES AND “TIPS-N-HINTS”
The following provides a brief overview of specific tasks and techniques that a Software Tester should learn or know while working on a game project.
1) File structures
A new software tester should get familiar with Radical’s file structure. There are several good documents pertaining to this from the global game and foundation technologies technical documents. The learning objective is basically to learn the overall game component architecture, and understand how it all flows together and to familiar yourself with the file dependencies of the game. When checking a game module, it is important to understand what parts its inner workings are dependent on. This will help you and speed up your bug investigation and problem analysis process.
2) Make Files
A software tester will be responsible for the nightly builds. Every night the build is built using the latest modules so that in the morning everyone can be working on the same up-to-date code. Occasionally some thing will go wrong with the nightly build. A file should generate when the build is broken and it should say what part it failed on. You should use this file to in your problem investigation. Usually some of the code may not work with another new part or maybe the pipelines are messed up. It is the software tester's responsibility to go through the broken module and to find out why it’s broken and fix it if possible. If its too much for you to fix, you should then inform the person responsible for the code or the pipeline that something is broken and get them to fix it. Once the problem is fixed the build should be rebuilt to verify. The build number should be automatically amended with a postfix to represent the build number of the day. These build numbers will help you track down bugs or identify bugs within a specific build. For example someone might come to you with a bug on an earlier version. If you can identify what build they are working on and what build is completed, it could just be a simple fix of them getting the new build for the bug fix. It would be a good idea to send an email that tells people a new build is completed.
Another note on build is to make sure to mark the build release or debug because there could be some problems with one but not the other.
3) Memory management
It is a good idea to check the current status of the memory after a nightly build. If there are significant changes between the new day and day before, make a note of it. Something could be wrong. If the exe size grows too large then the game will take longer to load.
4) Debugger and Code inspection
Allot of your time is going to be spent on debugging or at least helping someone debug the game. One of the main skills a software tester should have is a STRONG understanding on how a debugger (Visual C++, Linux Debugger) works so that they can accomplish this task. They should be able to step through or into parts of code and be able to understand what is happening and how to set and change variables as well as run a function at any time during the program.
At this time you will be doing some code inspection. The debugger can give you a lot of information on variables. Use this information to verify that the code is using all of its generated nodes and arrays and that any discarded arrays or nodes are properly cleaned up so no null pointer or such are created. Check the code for correct documentation for ease of use (e.g. no funky variables like temp;) ensure that the programmers are documenting correctly so their code can be picked up easily in the off case someone will have to continue their work.
You may have to also do some sort of data inspection depending on the game. If a game uses some sort of statistics program and randomization, make sure the data is being stored correctly and is only available to the parts of the program that use it. Ensure that the data is not getting corrupted as it is coveted and used in the pipelines.
5) Test Programs
A very important part of the job will be for a software tester to write test programs. Some examples would be programs to enable cheat codes, play sound files, enable/disable the AI, recreate test scenarios, and whatever other debug information may be required.
The Game Tester also uses these test programs so they can test for specific gameplay scenario and/or situations.
6) Sound Testing
Sound testing is simply listening to the sound and making sure the startup point and the length are appropriate. The software Tester often requires to write a “sound test” program to play all the individual sound files from inside the game (just a quick tip, you should check with the sound artist and see if they have one already).
Sound testing includes testing if there will be any error in loading the files, listening to the sound files for errors or distortion. Also, test to make sure all the sound files are being loaded correctly. If the game has Colour Commentary (or CC), use the cc profiler to analyze. The analysis helps you to determine how many times a CC was called, and if it gets played or overwritten.
The analyzer relates the scene or situations to the cc and checks to see if the information gets played, if you notice any problem in cc, you should check if the information is being passed and played correctly. Of course one way to analyze the cc is actually listening to it and note any irregularities and problems.
7) P3D and pipelines
It is important to generally understand how the pipelines work. How the sound is being processed and converted, how the Maya art is being converted and used, generally how all the object data is exported from a raw data to a useable product. It would be a good idea to understand how P3D files are created and used. Look into the Anim classes, P3D chunks and complex messes.
8) Database and Game Statistics
The main purpose of the database for a game is:
1) Player or team statistics. It could be used solely for display, or it could b used by the AI engine to simulate the actual game play or for randomization.
2) Run-time statistics. It is used by the game engine to remember certain state, attributes or scoring of the player as you move from level to level (or to a different game state).
The easiest way to verify the database is to go into the game itself using a debugger, and analyze the game is using the data correctly. Obtain a printout of the database and look at the screens to make sure the data is being loaded, in the right place, showing the right information. On a higher note, one could check the database export to make sure no error found in the export. Another good technique is to do random checking, say, on a player statistic, to make sure the data is correct.
9) Overlays
Make sure the overlays are loaded correctly. A tester could write a program to test that the overlays are being called correctly, and identify if any one does not get used at all.
10) Front End
Most of the Front End screens could relate to statistics (you know, in the database). The main thing to look into is the “Text Bible”, i.e. the appropriateness of the text, the naming convention, checking against any manufacture standards or requirements, and even spelling.
11) Bug Tracking Software
The software tester should know how to use PVCS Tracker to track and report bugs, and know the use of the standard template to create/customize a new defect database for a new project. It would be preferred if you could have administrator rights to tracker to set up others with accounts and manage the database (new client sign-on ID, fields and information).
12) Game Tester and love of the game
A software tester must also have “to have a love of games” in mind. A significant amount of your time will be playing games to find the bugs. You have to have the skills to find and report game bugs as well as help repair them. It is important to test for the fun factor of a game as well, if you stop having fun while playing the game then make sure to discuss with your peers like the Game Tester. If the changes happen too late they may not get implemented. Still, remember the debugger is your best friend.
13) Burning CDs
Simple, burn CD copies of the game for people who need them (game testers, management). A bigger part of this would be to track down problems of bugs being discovered on a different medium.
7. TESTING CONSIDERATIONS FOR SOFTWARE TESTING
7.1 TESTING COVERAGE
It will be way too much for a Software Tester to handle if someone has to do 100% of above. Unlike Game Testing, I believe some of these tasks are discretionary. You must learn and decide how much testing is enough. As a rule of thumb, the Software Tester should take a “best-effort based” approach and “time-box” his or her work and involvement for a game project. Once you come up with a test design and some high-level test plan, discuss with the game Technical Director to confirm the scope and priority.
7.2 TESTING VS. DEBUGGING
We all know good analysis of a bug simplifies the debugging process. However, there is a balance to strike between the amount of investigation and analysis done by a Software Tester and the amount done by a Developer. Here are some factors to consider:
§ Isolate the problem so it can be assigned to a single person in the bug report.
§ Describe the problem at the appropriate level of details so the developer can work on it without wasting too much time to re-investigate and explore the problem.
§ Are we in the “crunch” time – is the developer on the critical path?
§ Are you a more skilled debugger than the developer who is assigned for this bug? How we can get more productivity and better use of time?
7.3 TOOLS SUPPORT FOR TESTING
The Game Tester requires a fully functional Test Environment for testing. This test environment is comparable to a debugging environment that is used by the game developer or the Software Tester. Also, the test environment must support testing the graphic components of the game (e.g. providing some of the artwork run-time information; not just showing the artwork itself).
The Software Tester is expected to provide support and help out the Game Tester in using a new Test Environment platform.
8. FULL CYCLE TESTING
Game development goes through a cycle of stages. The game software is defined, evaluated, designed, built, tested, fixed, Alpha, Beta and then Final. The game development cycle involves many tasks by multiple functional teams, and in multiple stages. Most of the stages are often described sequentially but the underlying tasks are often proceeding in parallel, especially during game production. Testing and fixing can happen at any stages in the project cycle. However, the cost of fixing an error increases exponentially as project progresses.
The objective of “Full Cycle Testing” is to identify and fix the error (aka defect or bug) as early as possible so a quality product will be delivered at the end. In a simple terminology, the techniques used for “Full Cycle Testing” are 1) validation for accuracy and correctness, and 2) verification for completeness.
8.1 TESTING FOR DIFFERENT PROJECT STAGES
§ Requirement stage – game idea, storyboard, features and requirement document
§ Game Design stage – conceptual design of the new features
§ Technical Design stage – mainly, the Technical Design document and the Feature Specification. Also, the game architecture, session manager, front-end, AI, director, rendering, animation, frame overlay and so on. This encompasses most of the components assigned to the team and support teams
§ Development stage – coding and implementation/integration of various pipelines and work by the teams that are integrated to make up the game. “Clear box” testing will encompass module/component testing, testing of coverage and flow, data integrity, algorithm-specific testing (e.g. debug code is in the game code), path testing, and various levels of functional regression testing. This is more an incremental testing.
§ Review stage – Focus Group, and a “big bang” testing both Clear Box and Black Box Testing, in a structured manner.
§ Regression Testing stage – it will be the game testing (Alpha, Beta, and Final).
8.2 REVIEW CRITERIA AND CHECKLIST
Here are the testing criteria for the requirement phase and the design phase:
Requirement Document:
Are these the right requirements for the new game?
Are they complete?
Are they achievable?
Are they realistic?
Are they reasonable?
Are they compatible to other competitor game?
Are they testable?
Are there quantifiable and measurable targets?
Are they in line with our direction of the company (or company future)?
Game and Technical Design Document:
Is the design good?
Is the design complete?
Is the design doable?
Is the design equipped with the necessary error handling?
Does the design provide any debugging facility?
Is the design in line with our game architecture?
Has the design received review and approval?
9. TEST PLAN AND TESTING DOCUMENTATION
Effective testing relies on a complete Test Plan package. The document should be written in a way that promotes reusability (reuse a large portion of the previous test plan documents for a new game) and consistency (to minimize learning curve; the testing documentation produced for the game projects all have a similar format).
The hierarchical listing provided below is to outline a structure and the testing document framework that the testing will be handled for each game project.
§ Testing Requirements Document (as explained earlier)
§ Software Testing Test Plan
§ Game Testing Test Plan (positive testing – what is expected?)
§ Game Testing Scenario (negative testing – how robust is the game?)
§ Manufacture Platform Technical Requirements (e.g. SONY Technical Requirement Checklist for the PSX/PS2 game titles, and Microsoft Technical Requirement for the X-Box game titles)
The Game Testing Test Plan is a “positive testing” document that explains WHAT and HOW to test for the playability of the game (e.g. options and features).
QA has developed a guideline to explain the SONY requirements, section by section. We also reviewed the SONY Bug Report from our previous game submissions and develop a “our lesson learned” checklist so “we won’t make the same mistake twice”. As far as timing goes, the review for the manufacture standards and technical requirements is often conducted after Alpha.
In the Test Design document, you will learn the details (e.g. definition, guideline, templates and ideas) of preparing the test plan documents.
10. HOW TO CONDUCT EFFECTIVE TESTING
These are some of the “good ideas” from the experienced QA testers:
§ Make sure you are testing on the most up-to-dated version of the game. With at least one version possibly created on the same day, it is easy to find yourself testing the wrong version.
§ Make sure that every time you get a new game build, you are getting every part of a new game (a new game has more than one file!) and it may not work against a previous saved game from a Game Save/Load device like MEMORY card.
§ Keep track of your hours so you have time to look for new bugs and regression test the “fixed” bugs that you need to verify for closure.
§ A good understanding of the Tracker database is a MUST. You must understand the Tracker template, the definition and attributes of each field. An optional field does not mean the field is not required; the field is not required only if it is not applicable to the bug.
§ We must regression test the previous bugs. Just because something got fixed in the previous versions is not a guarantee that it still works in the current version. This is our mandate to re-test the previously closed bugs with the severity of A-Crash, B-Non functioning Feature, and C- Major Functional every week and for each major milestone.
§ Regression testing can also be frustrating, make sure you stay focused and don’t think of it as a trivial task.
§ It is NOT only tester who can enter bugs to the defect database. However, you should review the bug report if that is entered by a non-tester person (why? It is because you are responsible for regression testing, and the testers are responsible for all the bugs in the defect database).
§ Keep in mind that you will find bugs, however your productivity is not measured by the number of bugs that you find. There is no fixed rule for a game tester as the number of bugs that he or she must report in a day.
11. EXTERNAL TESTING
11.1 TESTING BY THE PUBLISHER
When a game is getting ready, we send the game to the publisher so they can take a closer look of the game and provide us with their feedback. Roughly 2 to 4 weeks before the Beta milestone date, we start sending the Publisher with the game build twice a week (e.g. every Tuesday and Friday). When the game is Beta, we will send the Publisher with our daily build of the game. Most of the publishers have their own FTP (File Transfer Protocol) site where we can upload the game file over the Internet. You should find out the FTP instructions from the Publisher.
The Publisher sends us their bug reports weekly to almost daily as the game is getting closed to Final. You must be aware that the publisher tests the game remotely, and uses a build that is a few days older than our latest version. We often receive “crack” bugs (i.e. misinterpretation of the feature) or bugs that have already been identified. This is the responsibility of the Test Lead to rectify these defect records before they are reviewed by the game designer or Producer. To facilitate the bug submission by the publisher, the Test Lead will provide with a file exchange format and the PVCS Tracker Defect Reporting template to the publisher so their bug data can be imported to our defect database.
11.2 TESTING BY THE MANUFACTURER
When the game is (or closed to) Final, the game is sent to the game platform manufacturer for testing. The primary focus of testing is on the peripherals and devices (e.g. controller and game pad), the naming convention, standards and the legal text appeared in the NIS or menus. The tester must be familiar with the Technical Requirements for each of the game platform.
Usually there are two kinds of submission to the manufacturer:
§ Pre-submission: The game is sent to the manufacturer (e.g. SONY, Microsoft) so they will briefly look at it and provide us with some feedback. Pre-submission gets us some idea about the readiness of the game. This is sort of an informal review, if we don’t receive any problem, it doesn’t necessarily mean the game is bug free. We often have a couple of pre-submissions for a brand new game (e.g. our first action game of Jackie Chan Stuntmaster), and we may skip the pre-submission for the second generation of a previous game title.
§ Submission: The game is formally reviewed by the manufacturer, usually a team of people is assigned to test the game. The result in the form of a manufacturer bug report is sent to us through the publisher. The submission process takes about a week. A game must meet certain standards and requirements (e.g. no crash bugs, no more than certain number of non-critical bugs identified by the manufacturer) before it is considered “commercial ready” by the manufacturer, and no game can be published without manufacturer’s approval.
There are certain rules that must be followed for the submission procedure, e.g. the use of special CD media, the number of copies, and the game packaging, etc. Please refer to the manufacturer specification for details. The Test Lead must work closely with the Producer, the Publisher and the testing group of the manufacturer for the submission process.