Testing Guidelines

General Rules

  • Tests are written using Pester.
  • Preferably use the concept of test-driven development when writing tests.
  • Each DSC module should contain the following Test folder structure:
       tests
       |---Unit
       |---Integration
  • The tests\Unit folder must contain a Test Script for each DSC Resource in the DSC Module with the filename DSC_<ResourceName>.Tests.ps1.
  • The tests\Integration folder should, whenever possible, contain a Test Script for each DSC Resource in the DSC Module with the filename DSC_<ResourceName>.integration.Tests.ps1.
  • Each Test Script should contain Pester based tests.
  • Integration tests should be created when possible, but for some DSC resources this may not be possible. For example, when integration tests would cause the testing computer configuration to be damaged.
  • Unit and Integration tests templates should be created using the Plaster template that is generated by the Sampler project.
  • The Unit and Integration test templates are dependent on the CI pipeline that is generated by the DSC module Plaster template that is generated by the Sampler project. That template contain a CI pipeline that when configured it will help get the dependencies to run the tests.

Creating DSC Resource Unit Test Instructions

Not yet written. This should be done through the Plaster template in the Sampler project.

Creating DSC Resource Integration Test Instructions

Not yet written. This should be done through the Plaster template in the Sampler project.

Updating DSC Resource Tests

It is possible that the Unit and Integration test templates may change in future, either to support new functionality or fix an issue. When a change is made to a template, it will be documented in the change log of the repository Sampler. To get the changes generate a new DSC module template from the Plaster template in the Sampler project.

DSC Resource HQRM Tests

The HQRM tests are present in the repository DSCResource.Test. The DSC module template will by default use the HQRM tests in this repository.

It is very likely that the DSC resource HQRM tests will be updated in the future and that will be documented in the change log of the repository DscResource.Test. The CI test pipeline defaults to opt-in to all tests so there are no need to do anything to get new functionality in the DSC modules.

Opt-Out from Tests

Default is that all the tests are opted-in. If a test cannot be resolved to pass then there are different scenarios supported to opt-out from tests. Preferably you should opt-out from tests.

  • Configuring the build.yaml with the tag ExcludeTag under either the key Pester or DscTest (JSON files override excludes for the key DscTest).
    Pester:
      ExcludeTag:
        - 'TagOnUnitTest'
    
    DscTest:
      ExcludeTag:
        - Common Tests - New Error-Level Script Analyzer Rules
        - Common Tests - Validate Example Files
        - Common Tests - Relative Path Length
    

Default opt-outs

Test “New Error-Level Script Analyzer Rules”

This HQRM tests is opt-out by default since it tests the ScriptAnalyzer rules PSDSCDscExamplesPresent and PSDSCDscTestsPresent which does not currently work with the folder structure needed by CI pipeline.

DscTest:
    ExcludeTag:
    - 'Common Tests - New Error-Level Script Analyzer Rules'

Even if opt-out this test will still run. You opt-out from the test phase failing if there are any violations. Noteworthy is also that if there are violations the opt-out test will be skipped (yellow), if there are no violations the test will pass (green).

Pinning version of prerequisites module

Not yet written.

Running Tests

If configured, all tests are automatically run using the CI pipeline via Azure Pipelines in the DSC Community Azure DevOps organization when updating the upstream repository.

Running Tests Locally

It is recommended that all tests be run locally before being committed.

Regardless of the method of running the tests you must always resolve the dependencies first. After that you must build the module.

  1. Resolve dependencies
  2. Build module

NOTE! Each time a source file changes you must re-build the module before running tests again. This is because the tests are always run against the built module in the ‘output’ folder, not the modules source files.

KNOWN ISSUE: If the DSC resources are importing a module from the folder Modules (common module), the common module is already imported if you run the tests a second time. That means that any changes that you make to the common module are not reflected in the test run. To workaround this the common module have to be manually removed using Remove-Module -Name 'ModuleName'. There are a second workaround, force import the module in the DSC resources. Force importing the module are not recommended due to performance and could have a big impact in production.

Running Tests Locally Using Pipeline

Remember to run the build task if any file has been changed in the source folder.

This runs all the tests for the module. BE AWARE: This may be configured to run integration tests too if they exist! Integration tests may disrupt the system configuration temporarily, leave system changed, or may require specific dependencies to execute.

.\build.ps1 -Tasks test

This will just run the unit tests for the module using code coverage.

.\build.ps1 -Tasks test -PesterScript 'tests/Unit'

This will just run the integration tests for the module without code coverage.

.\build.ps1 -Tasks test -PesterScript 'tests/Integration' -CodeCoverageThreshold 0
Running Tests Locally Using Invoke-Pester

Not yet written.

Running Tests Locally Within Visual Studio Code

Not yet written.

Running tests in free Azure DevOps organization

If you have Attached your fork to a free Azure DevOps organization then for each commit that is pushed to your working branch in your fork the pipeline will be run.

Re-run failed jobs

It is possible to re-run failed jobs in the Azure Pipeline. Be aware that in some circumstances it caches dependencies so even re-running a failed jobs that you know will pass it will not. Instead start a new job from the commit in question. If it is a pull request (PR) that failed push another change to the PR, or close and directly reopen the PR.

Unit Testing Private Functions

If a resource contains private (non-exported) functions that need to be tested, then to allow these functions to be tested by Pester, they must be contained in an InModuleScope Pester block:

InModuleScope $script:dscResourceName {
    Describe "$($script:dscResourceName)\Get-FirewallRuleProperty" {
        # Test Get-FirewallRuleProperty private/non-exported function

    }
}

Note: The core DSC Resource functions Get-TargetResource, Set-TargetResource and Test-TargetResource must always be public functions, so they do not need to be contained in an InModuleScope block.

Using Variables Declared by the Module in Tests

It is common for modules to declare variables that are needed inside the module, but may also be required for unit testing. One common example of this is the $script:localizedData variable that contains localized messages used by the resource. Variables declared at the module scope (private variables) can not be accessed by unit tests that are not inside an InModuleScope Pester block.

Note: See Localization section for more information of using localized messages in tests.

There are three solutions to this:

  1. Run the tests inside a InModuleScope which give access to all script scope variables like $script:localizedData.

  2. Use the InModuleScope to copy the private variable into a variable scoped to the Unit test:

    $localizedData = InModuleScope $script:dscResourceName {
        $script:localizedData
    }
    
  3. Add any variables that are required to be accessed outside the module by unit tests to the Export-ModuleMember cmdlet in the DSC Resource:

    Export-ModuleMember -Function *-TargetResource -Variables LocalizedData
    

Creating Mock Output Variables without InModuleScope

One useful pattern used when creating unit tests is to create variables that contain objects that will be returned when mocked cmdlets are called. These variables are often used many times over a single unit test file and so assigning each to a variable is essential to reduce the size of the unit test as well as improve readability and maintainability. For example:

$MockNetAdapter = @{
    Name              = 'Ethernet'
    PhysicalMediaType = '802.3'
    Status            = 'Up'
}

# Create a mock

Mock `
    -CommandName Get-NetAdapter `
    -MockWith { return $MockNetAdapter }

However, when InModuleScope is being used the variables that are defined won’t be accessible from within the mocked cmdlets. The solution to this is create a script block variable that contains the mocked object and then assign that to the Mock.

$GetNetAdapter_PhysicalNetAdapterMock = {
    return @{
        Name              = 'Ethernet'
        PhysicalMediaType = '802.3'
        Status            = 'Up'
    }
}

# Create a mock

Mock `
    -CommandName Get-NetAdapter `
    -ModuleName $script:ModuleName `
    -MockWith $GetNetAdapter_PhysicalNetAdapterMock

Example Unit Test Patterns

Pattern 1: The goal of this pattern should be to describe the potential states a system could be in for each function.

        Describe 'Get-TargetResource' {
            # TODO: Complete Get-TargetResource Tests...

        }

        Describe 'Set-TargetResource' {
            # TODO: Complete Set-TargetResource Tests...

        }

        Describe 'Test-TargetResource' {
            # TODO: Complete Test-TargetResource Tests...

        }

Pattern 2: The goal of this pattern should be to describe the potential states a system could be in so that the get/set/test cmdlets can be tested in those states. Any mocks that relate to that specific state can be included in the relevant Describe block. For a more detailed description of this approach please review #143

Add as many of these example ‘states’ as required to simulate the scenarios that the DSC resource is designed to work with, below a simple ‘is in desired state’ and ‘is not in desired state’ are used, but there may be more complex combinations of factors, depending on how complex your resource is.

Describe 'The system is not in the desired state' {
    #TODO: Mock cmdlets here that represent the system not being in the desired state


    #TODO: Create a set of parameters to test your get/set/test methods in this state

    $testParameters = @{
        Property1 = 'value'
        Property2 = 'value'
    }

    #TODO: Update the assertions below to align with the expected results of this state

    It 'Should return something' {
        Get-TargetResource @testParameters | Should -Be 'something'
    }

    It 'Should return false' {
        Test-TargetResource @testParameters | Should -Be $false
    }

    It 'Should call Demo-CmdletName' {
        Set-TargetResource @testParameters

        #TODO: Assert that the appropriate cmdlets were called

        Assert-MockCalled Demo-CmdletName
    }
}

Describe 'The system is in the desired state' {
    #TODO: Mock cmdlets here that represent the system being in the desired state


    #TODO: Create a set of parameters to test your get/set/test methods in this state

    $testParameters = @{
        Property1 = 'value'
        Property2 = 'value'
    }

    #TODO: Update the assertions below to align with the expected results of this state

    It 'Should return something' {
        Get-TargetResource @testParameters | Should -Be 'something'
    }

    It 'Should return true' {
        Test-TargetResource @testParameters | Should -Be $true
    }
}

To see examples of the Unit/Integration tests in practice, see the NetworkingDsc MSFT_DhcpClient resource:

Localization

When resources use localization (which all normally should) there is the possibility to use the localized messages in the tests. This is normally used to verify the correct error messages are thrown. But could be used for verbose messages, warning messages, and of course any other types.

When using the Get-LocalizedData helper function to load the localized strings into $script:localizedData in the resource module script file, the strings can easily be used like this in tests.

Note: For this to work the test (the It-block) must be inside a InModuleScope-block.

It 'Should throw the correct error message' {
    {
        Set-TargetResource @setTargetResourceParameters
    } | Should -Throw $script:localizedData.DatabaseMailDisabled
}

Helper functions for testing localization

There are two helper functions to simplify testing localized error messages. The helper functions can be used to build a correct localized error record to be used in the tests. They are used together with the Helper functions for localization.

These test helper functions can currently only be found in other resource modules, for example in the resource module NetworkingDsc, in the CommonTestHelper module. To use it, copy the PowerShell module CommonTestHelper.psm1 to the new resource module.

Get-InvalidArgumentRecord

This helper function returns an invalid argument exception error object. Below is an example of how it could look like in the tests to test that the correct localized error record is returned.

$errorRecord = Get-InvalidArgumentRecord `
    -Message ($script:localizedData.InterfaceNotAvailableError -f $interfaceAlias) `
    -ArgumentName 'Interface'

It 'Should throw an InterfaceNotAvailable error' {
    { Assert-ResourceProperty @testRoute } | Should -Throw $errorRecord
}
Get-InvalidOperationRecord

This helper function returns an invalid operation exception error object. Below is an example of how it could look like in the tests to test that the correct localized error record is returned.

$errorRecord = Get-InvalidOperationRecord `
    -Message ($script:localizedData.NetAdapterNotFoundError)

It 'Should throw the correct exception' {
    {
        $script:result = Find-NetworkAdapter -Name 'NoMatch'
    } | Should -Throw $errorRecord
}