Over the past few years the industry has experienced an increase in the pace of software development. More or improved features are available in subsequent generations of software products more rapidly than in the past. At the same time, businesses and individuals have more software to choose from to get the job done or for entertainment and information purposes. This poses some special challenges for teams working in software development. They must keep up with a fast pace of product development and because of the increased competitive landscape for software, the quality of their products must remain high.
The use of Agile methods when practiced with testing and fixing bugs early in the development cycle, otherwise known as continuous integration, have contributed to both the increased speed of software development and the improved quality of the final product. The use of tools to facilitate the automation of testing and collection/reporting of test results has also helped to increase speed and quality. Finally, team project management, particularly focused on quality practices, has a role to play in the speed and quality of software development. This series of blog entries will explore the use of the Agile method within a particular tool – Team Foundation Server (TFS) and focus on project quality management and continuous integration as key functions to increase the speed of software development and the production of higher quality products.
This first entry will review continuous integration practices, test automation within TFS and how they fit into the Agile process. It will also discuss the order in which to move efficiently to full use of continuous integration with the Agile process to manage quality.
The second blog entry will provide a mapping of quality management in the Project Management Body of Knowledge (PMBoK) to the Agile process within TFS. As Agile processes spread to larger organizations employing methods that reflect the PMBoK, this mapping can assist with the creation of a hybrid approach within TFS.
The final entry will describe our experiences bringing together aspects of project quality management, the Agile process and continuous integration in a real customer environment using TFS.
Continuous integration practices and test automation within TFS
We will review the use of build definitions, tests and work items such as user stories, tasks, test cases and bugs in TFS to facilitate continuous integration, test automation and quality management. Some basic information about these artifacts will be provided in this blog. However, more detailed information can be found at http://msdn.microsoft.com/en-us/library/dd380647.aspx.
Continuous integration practices and test automation within TFS begins with the use of build definitions, automated deployments and tests artifacts such as coded UI tests. Both build definitions and coded UI tests can be created and stored within TFS. Figure 1 below is an example of a build definition in TFS.
The deployment of built code to a test virtual machine (VM) is automated through the build definition.
Coded UI tests are created by using Visual Studio IDE or Microsoft Test Manager by recording the test during manual execution steps. These coded UI tests can then be stored and linked to a Test Case and/or Build Definition (Automated Tests – as defined by the Test.dll). Coded UI tests can run after the completion of a build and deployment of the code. Build and test agents running coded UI tests must be configured to run interactively.
Once the build and test artifacts have been created and stored in TFS, continuous integration can begin. The most common triggers for builds, deployments and tests are changes that are made by developers to the source code. Builds can be triggered as part of a gated check in policy, or by scanning the code stored in TFS for changes on a regular basis (e.g. hourly) and running a build. After either trigger for a build, tests run and a report is generated and sent to the relevant developers or team leads on the results. If something is broken, it can be fixed and checked in immediately. Figure 2 below is an example of a build report. This continuous integration practice incorporating automated builds, tests and issue reporting, keeps the code in a good, working state at all times.
To facilitate Agile development management, a process template is used within TFS. The process template defines the set of work item types (WITs), queries and reports that teams use to plan and track projects. We will focus on quality management, employing the user story, task, test case and bug work items in TFS. The user story work item is central to general management while the test cases, grouped into test suites and test plans are the key to quality management. The user story describes the functionality of an application – in Agile development it is the equivalent to user requirements.
Tasks can be created and linked to the user story to organize the development work. The Agile process task work item is shown below in Figure 4.
Work can be described in the details tab of the task. Source code can also be linked to the task, as can changesets, which are generated during code check in. These links facilitate traceability. At a higher level, tasks can be grouped by area and iteration and as is typical in Agile development management, iterations can be set up with start and end dates. This allows a view into how many tasks and how much work/coding/testing is planned for each iteration.
In Figure 5 below, steps are described in a test case along with the expected results.
The steps can be executed manually the first time and recorded creating a coded UI test and then linked to the test case (see tab on the test case work item called associated automation). Test cases can be grouped into a test suite (see Figure 6) through Microsoft Test Manager and also constrained by iteration – creating a goal for the completion of the test suite by the end of the assigned iteration.
The final WIT used in quality management in the Agile process is a bug. Failed tests can be used to generate bugs that are then assigned to the development team for resolution. Figure 7 below shows the detail that can be added to a bug, including links back to the test case that was executed to generate the bug. Screenshots, recordings of the issue and system information that point back to the VM or specific snapshot on which the issue was discovered, can help the development team reproduce the problem.
Bringing continuous integration within the Agile quality management process can be phased in over time. The artifacts required for continuous integration such as build definitions, deployments and test cases should be created first. Work items that link builds and test results back to changesets committed to source code, can then be used to organize and add traceability to a team’s work. The user stories and tasks are the foundation for the creation of test cases and grouping them into test suites. Adding test cases and plans later will allow for manual testing to elevate quality beyond what can be provided through continuous integration alone. Once all of the artifacts are set up in TFS and the team is using them, software development speeds up, while retaining a high level of quality.