Ty Balascio is principal author of the Windows End to End scenario testing template which I have provided in abbreviated form below.
Ty hopes to present metrics about E2E scenarios at STAR East 2006
The scenario is a story about someone (or something) trying to accomplish an action with the product under test. Tests of individual features and mechanical combination tests of related features or their input variables are not designed to provide this kind of check. Running scenarios within a sterile test lab practically validates product functionality in earlier more volatile builds. However, deploying scenarios to environments of real-users introduces platform and scenario variants which far surpass the quality of coverage possible within a ‘controlled test’ environment. In one sentence, End to End Scenarios testing is: “Validate that the provided solution meets the customer requirements by exercising real-world simulations of product usage”
Why is this category a test criterion?
The last few releases of Windows illuminated the fact that autonomous test teams struggled to replicate the plethora of scenarios encountered post-launch. A Functional testing
criterion attempts to cover these scenarios to a variable degree by some teams but not by all. Because of increased complexity, end scenarios tend to get neglected. Furthermore, teams become so acclimated with the look and feel (and workarounds) in a product that they tend to lose the “fresh eyes” perspective that larger audiences provide. Often product failures are not discovered until functionality from one component interacts with a separate, possibly unrelated component. The end result is an inconsistent user experience when we ship. The Shared Test Criteria’s answer to this gap in coverage is to formalize the approach taken with planning, executing, and measuring the deployment of user scenarios.
Who should be testing this category?
All product groups shall put thinking into the depth and quantity of scenarios needed in this area. While some groups only own back-end/infrastructure APIs, most teams own complete user scenarios in the box. As a general rule of thumb, the greater number of applications a product is dependent upon, the greater the need for e2e scenarios. The converse is also true; the greater the number of applications dependent upon a product, the greater the need for e2e scenarios. Therefore, a prerequisite for success within this test criterion is a strong and deep understanding of all integration points; which may include other MS teams or possibly external products.
||Mapping scenarios to personas
If you don’t know who your user is, then you are probably creating the wrong product. Personas
are a way of depicting the users of your product. Using personas lets you focus your tests at the right set of features to support the target user. Office, MSN, and Windows actively adopted personas in recent ship cycles as a mechanism to focus efforts on the right suite of features. Keeping this perspective enables the team to consider the skill-set of a SysAdmin, or the impatience of a boy game player when triaging issues. Here are some important questions to ask when thinking like a target persona:
- Why did this persona come to my component, and what are they here to accomplish?
- To what degree can the persona recover back to the task when they make a wrong turn? (are there any obvious trap doors?)
- Do features contain flows which map to the diverse skills of the intended audience?
1.4 When should we test this category?
Preparation for e2e testing begins before M0. The concentration of test passes will occur incrementally through milestones as appropriate features materialize. As feature sets stabilize, the passes will increase to reflect the supported scenarios.
Because she has a different perspective, the scenario tester will often do her own product and marketing research while she tests, on top of or independently of, research done by marketing and Program Management. Here are some useful ways to guide your research. It might seem that you need to know a lot about the system to use these and, yes, the more you know the more you can do. However, even if you’re new to the system, paying attention to a few of these as you learn the system can help you design interesting scenarios. Some sample work items would be to gather existing data from:
- Mining PSS tickets on previous versions to aggregate the top problematic scenarios.
- Soliciting feedback from partners for previous versions
- EEC data
- Beta participants
- Tuning in the “buzz”
- Usenet threads
- Magazine and WebZine reviews
- Web-pages covering the subject area
- Competitive analysis
Also, identify the other teams (MS or external) requiring integration with your product. During the preparation phase, the tester should know who the appropriate contact is, what level of participation is expected, and which areas of the product are within scope of the cross-group integration.
Scenario creation is usually done in conjunction with the due diligence underway within Program Management. Interestingly enough, merely executing the formal process of scenario definition becomes a catalyst for thinking about the details; resulting in the discovery of functionality gaps before the scenarios ever get run. During spec review, the proposed feature-set needs to be tested against this scenario draft to ensure that as many issues as possible are identified early. From a feature level, identify the solutions the feature provides, the possible users, and the usage pattern expected. Derive core scenarios, and scenario variants from these variables. PM/Dev/Test buyoff on core scenarios, implementation plan drafted, Verification checks
||Implementation and Execution Plan
At the core of e2e testing is the idea that disparate environments will introduce variables causing actionable issues to a product. For each scenario defined in 2.1.1, determine the most appropriate environment. Some factors to consider are:
- Simulates close to real world end user states.
- Participation quantities are large enough to satisfy measurement and scenario needs.
- Contains sufficient sampling of like configurations to accumulate trend data.
When considering the right environment, think about the frequency of the scenario, and the ease of setup. If it is a core and/or common task, then consider assigning the scenario to a self-host suite of tests which are ran a frequent intervals (see below). If scenarios necessitate complicated setups or repros, then consider delegating them to component testers who will be responsible for setup and configuration of the needed environment. If it is more efficient to automate the setups in a shared environment (such as a lab setup) then assign the scenarios accordingly.
This plan is complete when the following checklist is satisfied:
- Do all scenarios fit into a target environment?
- Are all target environments prepared?
- Permissions for integration acquired.
- Metrics measurement tools in place.
- Participation pool (users/integrating products) engaged
- Feedback mechanisms identified
- Execution tools readied
- Sample data
- Test applications
Test / Partners buyoff. Implementation plan complete.
Available MS Environments
Successful deployments and product integrations happen in ‘Dirty’ environments; pools of users where states fluctuate.
Available External Environments
is an authenticated Internet Web site providing shared infrastructure for products in their beta phase. Beta testers and/or guests log in with their .NET Passport ID/Password (http://www.passport.com
), and are presented with provided services. BetaPlace supplies product groups with a common framework for delivering technical beta information, provides a centralized beta source for beta testers, and increases beta administration efficiency.
What is the Enterprise Engineering Center
The Enterprise Engineering Center (EEC) is a facility for customers to recreate and test portions of their environments, meet with Microsoft product teams and give direct feedback about Microsoft products. To best impact product cycles, EEC engagements focus on pre-release products. Since customers test their deployment plans on replicas of their production environments, issues that might otherwise block deployment of our products are caught before those products are released. In what ways can Product teams be involved in a customer visit?
- Propose an EEC engagement (for critical customers or early adopters of your product): .
- Join DLs which will notify you of upcoming visits, kickoff meetings, and extended test opportunities.
- Attend engagement kick-off meetings: Customers typically give a short presentation at the beginning of their visit to explain their goals and test plans.
- Request to meet with customers: Many times customers would like to discuss new features, product design, best practices, frustrations, praises, and product roadmaps.
- Observe customers in the lab: Generally customers enjoy showing how they do things in production and discussing this with the people who developed and tested the code that they are using.
- Be available to troubleshoot customer issues: Occasionally customers will run into an issue that prevents them from deploying. You can volunteer to be available to troubleshoot issues and remove any deployment barriers.
- Get involved in extended test opportunities: Depending on interest and availability, the EEC can provide a customer’s environment for extended testing after an engagement has ended.