How to Run User Acceptance Testing using Azure DevOps?

Before a software product goes-live, during or at the end of the implementation phase, we tend to ask our users to perform a testing to check if the functionality meets the requirements and capture their feedback on bugs, issues or new change requests. This will make sure that the shipped features have the right quality and will be well-received by the userbase. Engineering teams follow various practices on how to conduct a UAT (User Acceptance Testing) end-to-end and in this article I would like to share with you our best practices which is indeed a continuous evolution till date and onwards.

Here is a quick checklist when planning a UAT:

1. Which user roles are involved and who will do the testing?
2. Which scenarios will be tested and what test cases will be executed for each scenario?
3. What tool(s) will be used by the users to access, execute and provide feedback for the test cases and by developers to access that user feedback and take necessary actions?
4. How will the overall test progress be monitored?

The primary focus of this article is the 3rd question on the tooling aspects but I will also briefly touch to the other points to show the steps end-to-end. The preparation of good quality scenarios & test cases is otherwise its own art-of-work and I will plan to cover that in one of the future articles.

I have seen and as well personally used many different tools to define and share the test cases such as online worksheets or SharePoint lists which work pretty fine to an extent; nevertheless it may take time to do the customization for the first setup (unless there are reusable templates) and more importantly, developers generally have their backlog somewhere else which will be disconnected from the user/tester backlog.

We are using Azure DevOps for the end-to-end product development where we maintain the backlog of all type of work-items (user stories, tasks, bugs …) and wanted to instead try to bring the users/testers to this experience vs having to maintain yet another repository for the testing assets. This approach enabled a connected experience across the testing team members and more productivity the way we perform UATs.

Next, I would like to walk you through step-by-step how to use Azure DevOps Test Suite for planning and running a UAT. In my scenarios, I will be using a fictional product “Dips Machine”. This is a magical device you can add the ingredients and it prepares you a variety of delicious dips to enjoy with your family & friends.

Photo by Creative Headline on Unsplash

1. Create Product Backlog

Let’s add some new features to our “Dips Machine” product.

Dips Machine Backlog

Story 1: As a user, I want to know the ingredients for Guacamole so that I can add them to the machine

Story 2: As a user, I want to prepare multiple portions with different spice levels so that I can offer them for my guests with different spice preferences

2. Create Test Plan

You need to select the test plan from the left menu on Azure DevOps and create a new one. (This feature is only available for the “basic + test plan” access level users. More information)

Create new test plan

3. Create Test Suites

Each test plan may have 1-to-many test suites. A test suite can be created in 3 ways: static, requirement based and query based. In our plans, we choose requirement based approach since such suites can be directly linked to user stories in the backlog which gives us a natural representation of our requirements in the test plan.

Create new test suite

Search for your backlog items to create a test suite for each

Test suites added to your test plan for each backlog item

Click on “View Requirement” to see the details of linked backlog item for the selected test suite

4. Create Test Cases

Now, you can add your test cases for each test suite. While writing the test cases, we generally take a direct reference to the acceptance criteria of the backlog items.

Click “New Test Case” to define your first test case

Add 1-to-many steps for each test case with action-expected result pairs

You can as well add attachments to each test step to give more instructions on how you want the users to take the actions. We generally attach the mock-ups or screenshots of the product screens to help users.

Test cases created for first suite (backlog item)

Test cases created for the second suite (backlog item)

5. Assign Test Cases

Once you define your test cases, next you need to assign them to testers in the team. In Azure DevOps taxonomy, test cases become test points when they are ready to be executed.

Assign test points to testers

If you want to assign multiple testers for each test point of the test suite, you can choose “Assign testers to run all tests” option and invite the testers.

Assign testers to run all tests

Search for users and assign test cases

6. Execute Test Points

Now the testers can see the test points assigned to them and can simply run them, go through each step, pass/fail them and raise bugs to the development team. Test points can be run using the desktop app or directly on the web browser. We choose the latter option as it requires no additional setup at testers’ desktops.

Run test case via web application

Pass/fail test steps, add comments on the failed steps, capture screenshots and create a bug

When creating a bug, the cool thing is that, the repro steps will be auto-generated into the description which helps the developers to troubleshoot the issue. Moreover, since the test suite was created as a “requirement-based suite”, this bug will be also automatically parented to the related backlog item (user story) and will appear in the sprint backlog.

Creating a bug for the failed step

7. Triaging Bugs

You can create a query to list the bugs created by your testers, triage them within the testing team and assign to developers.

Triage bugs

8. Monitor the UAT progress

You can generate charts to visualize your overall testing progress. If you want to monitor the progress for all the test suites, you need to create the chart at the root level as highlighted below.

Adding a chart at the root test suite level

Monitor the test result via test plan charts

I hope you enjoyed the article and it helped you to get a basic understanding of how Azure DevOps Test Plans can be used to end-to-end manage your user acceptance testing and enable a connected experience between testers and the developers. In my next articles, I plan to also talk about some advanced features of Azure Dev Ops Test Plans.