The Team Test Rig enables Team System users to run tests on one or more remote computers. A rig is made up of a single test controller and one or more test agents. The test controller can be installed on either one or two computers, and like Team Foundation Server, has both application and data tiers. A single test controller is used to coordinate the execution of multiple test agents running tests on multiple computers.

Supported Operating Systems

The following operating systems are supported for the agent and controller:

2008 release
Controller	Windows Server 2008 Windows Server 2003 (SP1) Windows Vista Windows XP Professional (SP2)
Agent	Windows Server 2008 Windows Server 2003 (SP1) Windows Vista Windows XP Professional (SP2)

 

2005 release
Controller	Windows Server 2003 (RTM) Windows XP Professional (SP2)
Agent	Windows Server 2003 (RTM) Windows XP Professional (SP2) Windows 2000 (SP4)

Hardware Requirements

For the test agent, memory and CPU typically control how much load can be generated. The test controller does not generate load, but does collect load test statistics from the agents and performance counter data from the agents and system under test. Therefore, database used by the controller requires considerable resources both for the amount of data stored and the number of agents involved in the load test scenario. Use the following table as a guide for determining hardware requirements for the controller and agent:

 

	Controller Application Tier		Controller Data Tier		Controller Application/Data Tier
	Min	Recommended	Min	Recommended	Min	Recommended
CPU	600 Mhz	1 GHz	600 Mhz	1 GHz	600 Mhz	1 GHz
Disk – system	1 GB	1 GB	1 GB	1 GB	1 GB	1 GB
Disk - install	1 GB	8 GB	8 GB	40 GB	8GB	48 GB
Memory	256 MB	512 MB	512 MB	1 GB	512 MB	1 GB

 

Agent	Minimum	Recommended
CPU	600 MHz	2 GHz
Disk - system	1 GB	1 GB
Disk - install	1 GB	5 GB
Memory	256 MB	1 GB

 

By default, SQL Express is installed with the controller and is used as the default store for load test results. Therefore, the controller runs an instance of SQL Express.  For more information, see “SQL Sizing Considerations” later in this document.

Supported Configurations

Visual Studio, the Test Controller, SQL, and the Test Agent can all reside on the same computer or be installed on different computers. Typical computer configurations are:

Recommended Configuration:

·         Computer 1: Visual Studio

·         Computer 2: Controller, SQL Express

·         Computer 3-n: Agents

Note   Installing an agent on the same computer as the controller can interfere with results collection. If you choose to do this, use the Agent Weighting property to reduce the amount of load generated by the agent installed on the same computer as the controller.

It is recommended that you install the agents on their own computer. Anything that is processing at the same time the agent is running affects the accuracy of the test. Agent weighting reduces the impact, but inaccuracies are still introduced.

The following additional configurations are available. They are listed in order of recommendation.

Alternative Configuration 1

·         Computer 1: Visual Studio, Controller, SQL Express

·         Computer 2-n: Agents

Alternative Configuration 2

·         Computer 1: Visual Studio

·         Computer 2: Controller, Agent

·         Computer 3: SQL Express

·         Computer 4-n: Agents

Alternative Configuration 3

·         Computer 1: Visual Studio

·         Computer 2: Controller, SQL Express, Agent

·         Computer 3-n: Agents

Performance Considerations

Here are some additional considerations to consider when sizing the hardware:

 

	Agent	Controller Application Tier	Controller Data Tier	Controller AT/DT
CPU	Depending on the test, the CPU is frequently the limiting factor	Not heavily used	Not heavily used	Not heavily used
Disk	Not heavily used	Not heavily used	10 GB space required for 24 hours of test data	10 GB space required for 24 hours of test data
Memory	Depending on the test, memory will be the limiting factor	Not heavily used	Heavily used by SQL	Heavily used by SQL

 

The amount of load that a specific load agent can generate varies widely from test to test. Most tests are bound by the CPU. CPU use is directly proportional to requests per second (RPS). For other load tests, memory is the limiting factor. The RPS you can expect to drive from a load agent depends upon many factors. These include the following:

·         User load

·         Think time

·         Authentication scheme

·         Size of requests and responses

·         Response time

·         Level of response validation

·         Test Type under load (either Web Test or Unit Test)

Think time is the primary factor for determining the number of users on a CPU-bound test. Changing think times from 2 seconds to 10 seconds allows you simulate 5 times more users, but the RPS being generated will be the same. If your goal is to simulate real users, set the think time to a value that reflects how you believe users will behave on your Web site. Increasing think time and the number virtual users will not necessarily place additional stress on your Web application.

Agents can be bound by memory on tests that use the Connection Per User connection mode. Two connection modes can be configured in the load test run settings. In Connection Pool mode (the default), connections are pooled, but each user still uses two connections when active. In this mode, all virtual users are multiplexed over the connection pool. This allows you to have 1000 active virtual users sharing 100 connections. In Connection Per User mode, each user has a connection that consists of two actual connections open to the server.

If load testing against a typical ASP.NET application with 3 to 5 second think time using Web tests, you can simulate around 1000 users from a single-processor load test agent with a 2 GHz CPU and 1 GB RAM (recommended configuration). The number of users supported is a function of the think time. With longer think times, more users can be supported.

SQL Sizing Considerations

By default, SQL Express is installed on the controller and is used by the controller as the default SQL store for load test results. The SQL Express database is license-limited to store 10 GB of data, which is around 24 hours of load test data for a typical load test. The space that is required for load test data varies greatly depending on the test.

During a load test, samples are collected for each counter instance on each computer. Therefore, the amount of space that is required in the database depends on the following factors: the number of counters collected, the number of computers involved in the test, and the number of samples taken, as controlled by the sample rate.

If appropriate, consider using a separate database to store the load test data. The database can be stored on either the controller computer or on a different computer. To change the data store, submit the SQL commands contained in the .sql file to the SQL server instance you want to use for the load test results store. These are two ways to do this. One way is to use the command sqlcmd from a command prompt and specify the options needed to connect to the desired database. Use the –i option to specify the path to loadtestresultsrepository.sql. Another way is to use one of the GUI interfaces to SQL, such as query analyzer, and open the .sql file and submit the connects.

Test Rig Software Requirements

Setup installs the following required components:

 

Component	Requirements	Version
Controller	SQL Express	2005
	.NET Framework	2.0
Agent	.NET Framework	2.0

 

Test Rig User Accounts

The following types of accounts exist in the rig. Before installing, create the required accounts on the rig computers.

 

Account Name	Description	Requirements
Setup User	User who installs the controller and agent.	This user must be an Administrator on the computer they are running Setup on, and be an Administrator on the Controller.
Controller Service account	User the controller is running as. During Setup, you are prompted for this user account.	This user must have access to read performance counters from computers under test during a load test because the controller service collects performance counters during a load test. If running in Workgroup (non-domain), a local computer account that has the same user name and password must exist on all agents. To restart the rig, the Controller Service User is added to the Agent Administrators group during Agent Setup.
Agent Service account	User the agent service is running as. During Setup, you are prompted for this user name.	By default, tests are run as this user. (There is a way to run tests as a different user). Therefore, this user needs the access as required by the tests you are running. If you turn on Agent Logging, this user needs write access to the installation directory.

 

Requirements for Workgroups

If the computers in a rig are running in Work Group mode, that is, they are not in a domain, or they are running in different non-trusted domains, create local computer accounts on the controller. The accounts should have a matching password for each user who will access the controller, including the Agent service users.

For example, you have three computers, Controller1, Agent1, and Agent2. You must create local computer accounts on each computer for the agent service user and controller user. You also must make sure an administrator account exists on all three computers with the same username and password.

 

Computer Name	User Account	Password
Controller1	ControllerService	ControllerServicePassword
Agent1	ControllerService	ControllerServicePassword
Agent2	ControllerService	ControllerServicePassword
Controller1	AgentService	AgentServicePassword
Agent1	AgentService	AgentServicePassword
Agent2	AgentService	AgentServicePassword

 

Note: If you are running on Windows XP in a work group, you must turn off Simple File Sharing. See this KB for info on how to do that: http://support.microsoft.com/kb/307874

Open Windows Explorer, choose Tools, and then choose Folder Options, Next, select the View tab, and turn off the "Use Simple File Sharing (Recommended)" option.

Accessing the Controller

After the controller is installed, access to the controller is limited to members of the TeamTestControllerUsers and TeamTestControllerAdmins groups that were created during Setup, and to the Administrators group. Add appropriate users and/or groups to these groups to allow them to access the controller. Members of the TeamTestControllerAdmins group or the Administrators group can administer the controller by clicking the Test menu, and then choosing Administer Test Controller in Visual Studio. Members of the TeamTestControllerAdmins group must also be power users or administrators on the controller computer.

Team Test Load Controller Installation

Warning   Set up the Controller before setting up any agents.

Note   Load controllers cannot be installed on domain controllers.

1.      Start setup.exe located in the vs\controller folder.

The Welcome to Setup page appears.

2.      Click Next.

The End-User License Agreement and Product Key page appears.

3.      Review the End-User License Agreement (EULA) and select I accept the terms of the license agreement, if appropriate.

4.      Enter the product key information and then click Next.

The Destination Folder page appears.

5.      Click Browse to select a different destination folder, or click Next to use the default folder.

The Service Account page appears.

6.      Enter the account name and password for the controller service account and then click Next.

The Ready to Install page appears.

7.      Review the installation information and then click Install to continue.

The Installing Components page appears.

8.      Upon successful installation, the Setup Completed Successfully page appears. Click Finish to complete the Setup program.

During Setup, you are prompted for the controller service account. The controller service account collects performance counters on other computers during a load test. Use a domain account that can easily be granted access to many computers. Grant the controller user rights to monitor performance on all the systems under test for the built-in performance monitoring capabilities to work.

Setup creates three local groups:

·         TeamTestControllerAdmins

·         TeamTestControllerUsers

·         TeamTestAgentService

Add users who run tests to TeamTestControllerUsers. Add users who administer the test rig to TeamTestControllerAdmins. Agent setup automatically adds the users who Agent Services run as to TeamTestAgentService.

Note   The user running the controller setup must be a member of local Administrators group.

The controller runs as a Windows Service on the computers on which it is installed. To view the properties of the service use Control Panel. In Control Panel, choose Administrative Tools and then choose Services. The name of the controller service is the Visual Studio Team Test Controller. The controller services can be started and stopped like other services in Windows.

The controller and agents can be configured and monitored by using the Administer Test Controller dialog box, available on the Test menu. For more information, see “How to: Administer the Rig” in the product documentation.

By default, the controller uses SQL Express to store load test results. SQL Express is license limited to store 4GB of data, which is approximately 10 hours of load test data. If you choose to use another SQL database to store the results, you can set up the Load Test Results Store after you have installed the controller. For more information, see “How to: Configure a Results Store using SQL” in the product documentation.

After you have configured your Load Test Results Store, you must indicate to the controller where the results are to be stored. For more information, see “How to: Identify a Load Test Results Store” in the product documentation.

Load Controller Firewall Exceptions

If you are running Windows Firewall, the controller setup adds the following exceptions to your firewall:

 

Exception	Reason
QTControllerService.exe	Enable communication with the controller
SQLExpr.exe	Enable Visual Studio Load Test Results Analyzer to access load test data stored in SQL Express
SQLBrowser.exe	Enable Visual Studio Load Test Results Analyzer to access load test data stored in SQL Express
File and Printer Sharing TCP 139	Enable Restart Rig in Administer Test Controllers dialog to work. Enable Agent setup to automatically add the Agent service user accounts to the TeamTestAgentService user group. Warning   Setup does not add this firewall exception. Add this exception manually.

 

If you are running on Windows XP in a work group, you must turn off Simple File Sharing. See this KB for info on how to do that: http://support.microsoft.com/kb/307874

Team Test Load Agent Installation

Warning    Set up the Controller before setting up any agents.

1.      Start setup.exe located in the vs\agent folder.

The Welcome to Setup page appears.

2.      Click Next.

The End-User License Agreement and Product Key page appears.

3.      Review the End-User License Agreement (EULA) and accept the terms of the license agreement, if appropriate.

4.      Enter the product key information and then click Next.

The Destination Folder page appears.

5.      Click Browse to select a different destination folder, or click Next to use the default folder.

Note   If you are installing the Agent on the same computer as the Controller, the option to choose a new location is disabled.

The Specify Controller page appears.

6.      Enter the computer name of the Team Test Load Controller computer and click Next.

Note   You can locate the name of the controller computer by clicking My Computer, right-clicking, choosing Properties and viewing the Computer Name tab.

The Ready to Install page appears.

7.      Review the installation information and then click Install to continue.

The Installing Components page appears.

8.      Upon successful installation, the Setup Completed Successfully page appears. Click Finish to complete the setup.

Following this, you will be prompted to restart your computer to complete the installation.

Note   The user running the agent setup must be a member of local Administrators group and a member of the controller’s Administrators group.

During setup, you are prompted for the name of the controller computer that this agent will be associated with. Agent setup communicates with the controller to add the agent to the controller.

During setup, you are prompted for the Agent user. This user is automatically added to TeamTestAgentService group on the controller. Only Agent Service Users who are a member of this group are allowed to communicate with the controller.

The Agent service is named Visual Studio Team Test Agent. To view the properties of the service use Control Panel. In Control Panel, choose Administrative Tools and then choose Services. The agent services can be started and stopped like other services in Windows. Also the agents can be administered using the Administer Test Controller dialog box, available on the Test menu. For more information, see “How to: Administer the Rig” in the product documentation.

Note   AgentConfigUtil.exe is a command line utility that can be used to assign an Agent to a different Controller, or to add an Agent to a Controller if Agent setup fails to do so.

Load Agent Firewall Exceptions

If you are running Windows Firewall, Agent setup added the following exceptions to your firewall:

 

Exception	Reason
QTAgentService.exe	Enable communication with the controller
AgentServiceUI.exe	Enable communication with the controller
QTAgent.exe	Enable communication with the controller
AgentConfigUtil.exe	Enable communication with the controller
File and Printer Sharing Port 139	Enable Restart Rig in Administer Test Controllers dialog to work Enable collection of performance counters Warning   Setup does not add this firewall exception. Add this exception manually.

 

If you are running on Windows XP in a work group, you will also need to turn off Simple File Sharing.

Troubleshooting

Note   To verify the installation, run Visual Studio and launch the Administer Test Controller dialog box (available from the Test menu), connect to the controller, and check the status of the agents. Agents should be in the Ready state. If an agent is not in the list of agents, reinstall the agent or run AgentConfigUtil.exe after verifying that the controller is installed correctly. If the agent is in the Disconnected state, check that the agent service user account has been added to the TeamTestAgentService user group on the controller. Also check the <installdir>\Microsoft Visual Studio 2005 Team Test Load Agent\LoadTest\agentConfigUtilSetup*.log file for errors.

“Rig failed to restart for controller <controller computer name>. The following error was reported Cannot open VSTTController service on computer <controller computer name>.”

This error occurs when a user tries to restart the rig from the Administer Test Controller dialog box, but while this user has been added to the TeamTestControllerAdmins group, they have not also been added to the Power User or Administrator group.