Using Azure Pipelines at Dailymotion for the UWP Team, Running UI tests with WinAppDrivers in Azure Pipelines with YAML [Part 3]

This article will talk how the dailymotion UWP team used Azure Pipelines to run UI test. We will go over how to setup this CI using YAML.

It will be divided into 4 parts:

Part 1 : Continuous integration, Creating automated Builds (x64, x86, Release, Debug, etc) using YAML

Part 2: Running Unit Tests in Azure Pipelines

Part 3: Running UI test with WinAppDrivers in Azure Pipelines

Part 4: Releasing test builds and publishing to the UWP Store

UI testing

To start out you might wish to read up on the Microsoft wiki about UI testing.

Project Structure

Here is the structure of the dailymotion solution:

build/
├── pipelines/ 
    ├──azure-pipelines.ci.yml
    ├──azure-pipelines.release.yml
    └──templates/
       ├──build-app.yaml
       ├──build-single-architecture.yaml
       ├──run-ui-tests.yaml
       └──run-unit-tests.yaml
  
└──scripts/ 
     └──UpdateAppxManifestVersion.ps1
src/
├── Dailymotion.AppOnly.sln
└── Dailyotion.*[Folders]
tests/
├── DailymotionUnitTests/DailymotionUnitTests.csproj
├── DailymotionUnitTests/DailymotionUnitTests.Core.csproj
└── DailymotionUITests/DailymotionUITest.csproj
Dailymotion.AllProjects.sln
README.md

Making WinAppDriver work with Azure Pipelines

First we are going to need to add the task WinAppDriver to our organisation: https://marketplace.visualstudio.com/items?itemName=WinAppDriver.winappdriver-pipelines-task

Also, you can get the task from the marketplace under the Tasks tab in Azure Pipelines:

Once you have gone through this step you can start updating you YAML file with the task :

- task: WinAppDriver.winappdriver-pipelines-task.winappdriver-pipelines-task.Windows Application Driver@0

Which will start your UI testing in your application.

Our YAML File for UI tests

Now we will look at our azure-pipelines.ci.yml file that will call run-ui-tests.yaml (which is a template file) which will execicute our UI tests:

# Universal Windows Platform
trigger:
- dev/*
pr:
- dev/*
name: 0.$(Date:yyMM).$(DayOfMonth)$(Rev:rr).0
# because we dont have a lot of resources  today only build in x64
jobs:
- template: ./templates/build-app.yaml
  parameters:
    platform: x64
    buildPlatform: Debug
# make sure WinAppDriver is activated in the AZURE PIPELINES org
- template: ./templates/run-ui-tests.yaml
  parameters:
    platform: x64
    buildPlatform: Debug
# Unit test for UWP project and Core libraries
- template: ./templates/run-unit-tests.yaml
  parameters:
    platform: x64
    buildPlatform: Debug

Every time a commit is done to one of our development branches we will build and run the UI test.

Next, we can look at our run-ui-tests.yaml file:

# This template contains jobs to run UI tests using WinAppDriver.
parameters:
  platform: ''
  buildPlatform: ''
jobs:
- job: UITests${{ parameters.platform }}
  displayName: UITests ${{ parameters.platform }}
  dependsOn: Build${{ parameters.platform }}
  condition: succeeded()
  pool:
    vmImage: windows-2019
  variables:
    skipComponentGovernanceDetection: true
  steps:
  - checkout: none
- task: DownloadBuildArtifacts@0
    displayName: Download AppxBundle and DailymotionUITests
    inputs:
      artifactName: drop
      itemPattern: |
        drop/${{ parameters.buildPlatform }}/${{ parameters.platform }}/Dailymotion.Neon/AppPackages/**
        drop/${{ parameters.buildPlatform }}/${{ parameters.platform }}/publish/**
- task: PowerShell@2
    displayName: Install certificate
    inputs:
      filePath: $(Build.ArtifactStagingDirectory)/drop/${{ parameters.buildPlatform }}/${{ parameters.platform }}/Dailymotion.Neon/AppPackages/Dailymotion.Neon_$(Build.BuildNumber)_${{ parameters.buildPlatform }}_Test/Add-AppDevPackage.ps1
      arguments: -CertificatePath $(Build.ArtifactStagingDirectory)/drop/${{ parameters.buildPlatform }}/${{ parameters.platform }}/Dailymotion.Neon/AppPackages/Dailymotion.Neon_$(Build.BuildNumber)_${{ parameters.buildPlatform }}_Test/Dailymotion.Neon_$(Build.BuildNumber)_x86_x64_arm_Debug.cer -Force
- task: PowerShell@2
    displayName: Install app
    inputs:
      filePath: $(Build.ArtifactStagingDirectory)/drop/${{ parameters.buildPlatform }}/${{ parameters.platform }}/Dailymotion.Neon/AppPackages/Dailymotion.Neon_$(Build.BuildNumber)_${{ parameters.buildPlatform }}_Test/Add-AppDevPackage.ps1
      arguments: -Force
- task: WinAppDriver.winappdriver-pipelines-task.winappdriver-pipelines-task.Windows Application Driver@0
    displayName: 'WinAppDriver - Start'
    inputs:
      AgentResolution: 1080p
- task: VSTest@2
    displayName: Run DailymotionUITests
    inputs:
      testAssemblyVer2: $(Build.ArtifactStagingDirectory)/drop/${{ parameters.buildPlatform }}/${{ parameters.platform }}/publish/DailymotionUITest.dll
      vsTestVersion: 16.0
      runSettingsFile: $(Build.ArtifactStagingDirectory)/drop/${{ parameters.buildPlatform }}/${{ parameters.platform }}/publish/DailymotionUITest.runsettings
      platform: ${{ parameters.platform }}
      configuration: ${{ parameters.buildPlatform }}
 
  - task: WinAppDriver.winappdriver-pipelines-task.winappdriver-pipelines-task.Windows Application Driver@0
    displayName: 'WinAppDriver - Stop'
    inputs:
      OperationType: Stop

The important tasks are:

- task: DownloadBuildArtifacts@0

To get the binaries of your application and the UI tests

- task: PowerShell@2

To install the UWP app and its certificates

-task: WinAppDriver.winappdriver-pipelines-task.winappdriver-pipelines-task.Windows Application Driver@0

to start running the WinAppDriver (just like when you are running the app on your laptop)

- task: VSTest@2

To execute the Tests

- task: WinAppDriver.winappdriver-pipelines-task.winappdriver-pipelines-task.Windows Application Driver@0

And lastly to stop the WinAppDrivers.

If you don’t want to use YAML to setup your WinAppDriver UI test you can also use the UI by following this sample:

microsoft/WinAppDriver

And that was a quick overview of how we were able to set up our Unit Tests when a commit is done on one of our development branch, in the next parts we will see:

Part 4: Releasing test builds and publishing to the UWP Store

Happy coding