This assumes the repository is using the pattern from the Sampler project. Also make sure you have updated the repository to the latest pipeline.
With the release of Pester v4.10 it is now possible to upload the Pester generated JoCoCo file. The same file is used to upload code coverage to both Azure Pipelines and to Codecov.io.
The DSC Community GitHub organization has already added the Codecov GitHub App on all existing repositories in the organization so that existing and any new repository will have the Codecov GitHub App added automatically. There is nothing that needs to be added to use code coverage in Azure Pipelines.
Information: Due to how Pester generates the JaCoCo code coverage file Azure Pipelines code coverage will not find source files in certain circumstances. When a repository is using the pattern from the Sampler project then sometimes the paths are just copied (not built) from source by ModuleBuilder. For example MOF-resources do show coverage for the entire file, but not on individual code lines since the source file can not be found by the task PublishCodeCoverageResults@1. Azure Pipelines code coverage expects the full relative path to be in the
<sourcefile>element, relative from the path specified in argumentpathToSourcesof the pipeline task PublishCodeCoverageResults@1. Pester does not do this. Codecov.io is smarter in that sense and builds the relative path from both the<package>and<sourcefile>element, and Codecov.io expects the<package>and<sourcefile>element to together match the source folder structure in the GitHub repository (at the commit). Azure Pipelines code coverage generate source files form the code that are available in the pipeline, and it can only be a single path, and that single path does not support pattern matching.
To upload code coverage we need to:
build.yamlcodecov.yml (if Codecov.io should be used)Test in the file azure-pipelines.yml by modifying
the existing unit test jobThere are additional steps if code coverage should be gathered from multiple jobs, see section Code coverage for multiple jobs.
If the repository is not building any part of the module, that is not
using the ModuleBuilder pattern of Private, Public, Classes and or
Enum. E.g. the repository is only using MOF-based resources that are
copied by ModuleBuilder.
build.yamlBecause Codecov.io expects the file that is uploaded
to be prefixed with JaCoCo we have to change the filename of the JaCoCo
test results file that Pester is creating. The file must also be created
with the encoding UTF8 (without BOM) so that Codecov.io can accept it,
so we make sure to change the encoding to ascii.
Under the key Pester, add the keyword CodeCoverageThreshold. CodeCoverageOutputFile,
and CodeCoverageOutputFileEncoding. They keyword CodeCoverageThreshold
must be set to a value between 1 and 100. Normally the value 80 (80%)
is a good value if the repository have enough coverage, lower the value if
it doesn’t.
It can look something like this:
Pester:
OutputFormat: NUnitXML
ExcludeFromCodeCoverage:
- Modules/DscResource.Common
Script:
- tests/Unit
ExcludeTag:
Tag:
CodeCoverageThreshold: 80
CodeCoverageOutputFile: JaCoCo_coverage.xml
CodeCoverageOutputFileEncoding: ascii
NOTE: the filename can be anything after the prefix
JaCoCo, for example it is possible to useJaCoCo_$OsShortName.xmlwhich results in the filenameJaCoCO_macOs.xmlwhen the test task is run on macOS.
azure-pipelines.ymlFrom the job Test_Unit we can remove the task Set Environment Variables
and the task Publish Code Coverage. Those two task will be moved to the
new job. Instead we add the task Publish Test Artifact which uploads
the test result files that ended up in the folder output/testResults.
At the top of the file, at the same level as the key trigger add the
following.
variables:
buildFolderName: output
buildArtifactName: output
testResultFolderName: testResults
testArtifactName: testResults
sourceFolderName: source
It should look something like this:
trigger:
branches:
include:
- main
paths:
include:
- source/*
tags:
include:
- "v*"
exclude:
- "*-*"
variables:
buildFolderName: output
buildArtifactName: output
testResultFolderName: testResults
testArtifactName: testResults
sourceFolderName: source
Test_UnitThe main tasks of this job must be:
DownloadPipelineArtifact@2 (or
use the same task that Build stage used)output/testResults (make sure CodeCoverageThreshold has a value
higher than 0)output/testResults folder to the artifact testResults
using the task PublishPipelineArtifact@1.Most important here is the task Publish Test Artifact is updated to
use PublishPipelineArtifact@1, which is expected to be able to download
the artifact in the code coverage job (see next section). The
Publish Test Artifact must be run after the test task.
The arguments for the task Run Unit Test can differ depending on repository.
But most important is if the CodeCoverageThreshold argument is used to
override the value in build.yaml then the value for CodeCoverageThreshold
may not be set to 0. The value 0 means that no coverage is gathered.
This is how it can look like:
- job: Test_Unit
displayName: 'Unit'
pool:
vmImage: 'windows-2019'
timeoutInMinutes: 0
steps:
- task: DownloadPipelineArtifact@2 # Must be present for build task to work.
displayName: 'Download Pipeline Artifact'
inputs:
buildType: 'current'
artifactName: $(buildArtifactName)
targetPath: '$(Build.SourcesDirectory)/$(buildArtifactName)'
- task: PowerShell@2 # Runs the tests, and generates code coverage.
name: test
displayName: 'Run Unit Test'
inputs:
filePath: './build.ps1'
arguments: "-Tasks test -PesterScript 'tests/Unit'" # <--- Arguments can differ depending on repository.
pwsh: false
- task: PublishTestResults@2 # <--- Task optional, not necessary for code coverage.
displayName: 'Publish Test Results'
condition: succeededOrFailed()
inputs:
testResultsFormat: 'NUnit'
testResultsFiles: '$(buildFolderName)/$(testResultFolderName)/NUnit*.xml'
testRunTitle: 'Unit'
- task: PublishPipelineArtifact@1 # <--- This task is most important.
displayName: 'Publish Test Artifact'
inputs:
targetPath: '$(buildFolderName)/$(testResultFolderName)/'
artifactName: $(testArtifactName)
parallel: true
CodeCoverageAdd a new job that depends on the job Test_Unit (see previous section)
since we must wait for the JaCoCo XML file to exist. The reason for having
a separate job is that we need (or at least it is easiest) to run the
Codecov.io upload task in a Linux build worker.
The tasks of this job must be:
DownloadPipelineArtifact@2 (or
use the same task that Build stage used)testResults that was upload by the job
in the previous section) - job: Code_Coverage
displayName: 'Publish Code Coverage'
dependsOn: Test_Unit
pool:
vmImage: 'ubuntu 16.04'
timeoutInMinutes: 0
steps:
- task: DownloadPipelineArtifact@2
displayName: 'Download Pipeline Artifact'
inputs:
buildType: 'current'
artifactName: $(buildArtifactName)
targetPath: '$(Build.SourcesDirectory)/$(buildArtifactName)'
- task: DownloadPipelineArtifact@2
displayName: 'Download Test Artifact'
inputs:
buildType: 'current'
artifactName: $(testArtifactName)
targetPath: '$(Build.SourcesDirectory)/$(buildFolderName)/$(testResultFolderName)'
- task: PublishCodeCoverageResults@1
displayName: 'Publish Code Coverage to Azure DevOps'
inputs:
codeCoverageTool: 'JaCoCo'
summaryFileLocation: '$(Build.SourcesDirectory)/$(buildFolderName)/$(testResultFolderName)/JaCoCo_coverage.xml'
pathToSources: '$(Build.SourcesDirectory)/$(sourceFolderName)/'
- script: |
bash <(curl -s https://codecov.io/bash) -f "./$(buildFolderName)/$(testResultFolderName)/JaCoCo_coverage.xml"
displayName: 'Publish Code Coverage to Codecov.io'
codecov.ymlThis file is not necessary if Codecov.io is not used.
NOTE: Make sure to update the default branch name in the Codecov.io project site if the default branch name is renamed.
NOTE: If this file exist and starts with a full stop
., e.g..codecov.yml, then please rename it tocodecov.yml. See this FAQ for more information https://docs.codecov.io/docs/codecov-yaml#section-can-i-name-the-file-codecov-yml
These settings can be set as desired, but the values below are what are used by default in the DSC Community repositories.
The important part is the key fixes. Codecov.io
is expecting the paths in the JaCoCo file to match the folder structure in
the GitHub repository. Make sure to change the part ::source to the name
of the source folder in the repository. Normally this is source, but the
repository can also use src or a folder name with the same name as the
module name.
Since the Sampler project runs
tests against the built module in the output folder, the paths do not
match those of the repository. The key fixes converts the paths in
the JaCoCo XML file to the correct paths that Codecov.io
expects.
See the codecov.yml Reference for more information about these settings.
codecov:
require_ci_to_pass: no
# main should be the baseline for reporting
branch: main
comment:
layout: "reach, diff, flags, files"
behavior: default
coverage:
range: 50..80
round: down
precision: 0
status:
project:
default:
# Set the overall project code coverage requirement to 70%
target: 70
patch:
default:
# Set the pull request requirement to not regress overall coverage by more than 5%
# and let codecov.io set the goal for the code changed in the patch.
target: auto
threshold: 5
fixes:
- '^\d+\.\d+\.\d+::source' # move path "X.Y.Z" => "source"
README.mdFinally we should add the status badges to the README.md. Replace
{repositoryName} with the actual repository name, e.g. ComputerManagementDsc.
Also replace {defaultbranch} to the name of the default branch, e.g. main.
[](https://codecov.io/gh/dsccommunity/{repositoryName})
If the repository is building all or just part of the module using the ModuleBuilder’s
pattern of Private, Public, Classes and/or Enum. E.g. combination of
class-based and MOF-based resources, and/or public/private functions.
build.yamlUpdate build.yml the same way as in the section Modify build.yaml
in Not building module, just copies files.
Add the task Convert_Pester_Coverage to the build
task test. This task will convert the line numbers from the built module
to the correlating line in the specific source file. The task will generate
a new coverage file that is then merged with the original file that Pester
generated.
test:
- Pester_Tests_Stop_On_Fail
- Convert_Pester_Coverage
- Pester_if_Code_Coverage_Under_Threshold
azure-pipelines.ymlUpdate azure-pipelines.yml exactly the same way as in the section Modify azure-pipelines.yml
in Not building module, just copies files.
codecov.ymlRead more about this file in the section Add codecov.yml in Not building module, just copies files.
This file is not necessary if Codecov.io is not used.
The build task Convert_Pester_Coverage will update the coverage file with
the correct name of the source folder. So when using build task Convert_Pester_Coverage
the covecov.yml should look like below. Note that the main difference
(from Not building module, just copying files)
is that it is not using fixes keyword as the task Convert_Pester_Coverage
will do that for us.
codecov:
require_ci_to_pass: no
# main should be the baseline for reporting
branch: main
comment:
layout: "reach, diff, flags, files"
behavior: default
coverage:
range: 50..80
round: down
precision: 0
status:
project:
default:
# Set the overall project code coverage requirement to 70%
target: 70
patch:
default:
# Set the pull request requirement to not regress overall coverage by more than 5%
# and let codecov.io set the goal for the code changed in the patch.
target: auto
threshold: 5
README.mdUpdate README.md exactly the same way as in the section Add status badge to README.md
in Not building module, just copies files.
If a repository needs to gather code coverage from more than one job, the code coverage files need to be merged before publishing them to either or both services Azure DevOps or Codecov.io.
For example, unit tests or integration tests are run on multiple operating systems and/or target multiple versions of an application. It might be that some functions can only be tested on a certain version/operating system while other functions can only be tested on a different version/operating system.
Start of by implementing the steps in either Not building module, just copying files or Building whole or part of module depending on the repository’s need.
build.yamlThe filename JaCoCo_coverage.xml that has been used in the previous section
cannot be used for Pester when we need to gather coverage from multiple jobs.
There are two options:
CodeCoverageOutputFile entirely so that the
default value is used. The default value names the files using the PowerShell
version and operating system (Linux, macOS, Windows) and with the prefix Codecov_
(short for ‘Code Coverage’).Depending on your choice, the code coverage job in Azure Pipelines needs to take account for it. More on that later.
If you choose to use a specific filename the Pester section can look like
this:
NOTE: the filename can be anything after the prefix
JaCoCo, for example it is possible to useJaCoCo_$OsShortName.xmlwhich results in the filenameJaCoCO_macOs.xmlwhen the test task is run on macOS.
Pester:
OutputFormat: NUnitXML
ExcludeFromCodeCoverage:
- Modules/DscResource.Common
Script:
- tests/Unit
ExcludeTag:
Tag:
CodeCoverageThreshold: 80
CodeCoverageOutputFile: JaCoCo_Merge.xml
CodeCoverageOutputFileEncoding: ascii
If you choose to remove CodeCoverageOutputFile keyword, the Pester section
can look like this:
Pester:
OutputFormat: NUnitXML
ExcludeFromCodeCoverage:
- Modules/DscResource.Common
Script:
- tests/Unit
ExcludeTag:
Tag:
CodeCoverageThreshold: 80
CodeCoverageOutputFileEncoding: ascii
We also need to add a new build task that we call merge that will run
the task Merge_CodeCoverage_Files. This task should be added under the
keyword BuildWorkflow:.
It can look something like this:
BuildWorkflow:
'.':
- build
- test
merge:
- Merge_CodeCoverage_Files
The task Merge_CodeCoverage_Files also needs to have the keyword
CodeCoverageMergedOutputFile and CodeCoverageFilePattern settings
defined in the build.yaml file.
The keyword CodeCoverageMergedOutputFile should be set to the filename
that the task Merge_CodeCoverage_Files will generate and is the file
that will be uploaded to the code coverage services.
NOTE: For the service Codecov.io the filename must be prefixed with ‘JaCoCo’.
The keyword CodeCoverageFilePattern is the pattern to recursively look
for under the output/testResults folder. It should use a pattern that
can recognize the JaCoCo files that the test jobs generate.
If using the default filenames it can look something like this:
CodeCoverage:
CodeCoverageMergedOutputFile: JaCoCo_coverage.xml
CodeCoverageFilePattern: Codecov*.xml
If a specific filename was used it can look something like this:
CodeCoverage:
CodeCoverageMergedOutputFile: JaCoCo_coverage.xml
CodeCoverageFilePattern: JaCoCo_Merge.xml
azure-pipelines.ymlFrom the job Test_Unit we can remove the task Set Environment Variables
and the task Publish Code Coverage. Those two tasks will be moved to the
new job. Instead we add the task Publish Test Artifact which uploads
the test result files that ended up in the folder output/testResults.
We can remove the global variable testArtifactName since it will not be used.
After removing the global variable it should look something like this:
trigger:
branches:
include:
- main
paths:
include:
- source/*
tags:
include:
- "v*"
exclude:
- "*-*"
variables:
buildFolderName: output
buildArtifactName: output
testResultFolderName: testResults
sourceFolderName: source
At least two test tasks must be used. The main tasks for each job must be:
DownloadPipelineArtifact@2 (or
use the same task that Build stage used)output/testResults (make sure CodeCoverageThreshold has a value
higher than 0)output/testResults folder to a, for each job, unique artifact
name using the task PublishPipelineArtifact@1.Most important here is the task Publish Test Artifact is updated to
use PublishPipelineArtifact@1, which is expected to be able to download
the artifact in the code coverage job (see next section). The
Publish Test Artifact must be run after the test task. The artifact name
must be unique for each test job.
The arguments for the task Run Unit Test can differ depending on repository.
But most important is if the CodeCoverageThreshold argument is used to
override the value in build.yaml then the value for CodeCoverageThreshold
may not be set to 0. The value 0 means that no coverage is gathered.
This is how it can look like:
- job: test_windows_core # Run tests in PowerShell 7 on Windows
displayName: 'Windows (PowerShell Core)'
timeoutInMinutes: 0
pool:
vmImage: 'windows-2019'
steps:
- task: DownloadPipelineArtifact@2 # Download build artifact.
displayName: 'Download Pipeline Artifact'
inputs:
buildType: 'current'
artifactName: $(buildArtifactName)
targetPath: '$(Build.SourcesDirectory)/$(buildArtifactName)'
- task: PowerShell@2 # Run tests and gather code coverage.
name: test
displayName: 'Run Tests'
inputs:
filePath: './build.ps1'
arguments: '-tasks test'
pwsh: true
- task: PublishTestResults@2 # Optional. Publish test results.
displayName: 'Publish Test Results'
condition: succeededOrFailed()
inputs:
testResultsFormat: 'NUnit'
testResultsFiles: '$(buildFolderName)/$(testResultFolderName)/NUnit*.xml'
testRunTitle: 'Windows Server Core (PowerShell Core)'
- task: PublishPipelineArtifact@1 # Publish test artifact with unique name for this job.
displayName: 'Publish Test Artifact'
inputs:
targetPath: '$(buildFolderName)/$(testResultFolderName)/'
artifactName: 'CodeCoverageWinPS7' # An unique artifact name.
parallel: true
- job: test_linux # Run tests in PowerShell 7 on Linux
displayName: 'Linux'
timeoutInMinutes: 0
pool:
vmImage: 'ubuntu 16.04'
steps:
- task: DownloadPipelineArtifact@2 # Download build artifact.
displayName: 'Download Pipeline Artifact'
inputs:
buildType: 'current'
artifactName: $(buildArtifactName)
targetPath: '$(Build.SourcesDirectory)/$(buildArtifactName)'
- task: PowerShell@2 # Run tests and gather code coverage.
name: test
displayName: 'Run Tests'
inputs:
filePath: './build.ps1'
arguments: '-tasks test'
- task: PublishTestResults@2 # Optional. Publish test results.
displayName: 'Publish Test Results'
condition: succeededOrFailed()
inputs:
testResultsFormat: 'NUnit'
testResultsFiles: '$(buildFolderName)/$(testResultFolderName)/NUnit*.xml'
testRunTitle: 'Linux'
- task: PublishPipelineArtifact@1 # Publish test artifact with unique name for this job.
displayName: 'Publish Test Artifact'
inputs:
targetPath: '$(buildFolderName)/$(testResultFolderName)/'
artifactName: 'CodeCoverageLinux' # An unique artifact name.
parallel: true
CodeCoverageAdd a new job that depends on all the test jobs (see previous section) since we must wait for all of the JaCoCo XML files to exist.
The tasks of this job must be:
DownloadPipelineArtifact@2 (or
use the same task that Build stage used)If each test job is using that same filename (specified in the build.yaml),
or that there is a risk that the default filename might not be unique, then
we need to specify specific folders where the test artifacts are downloaded.
The publishing tasks must specify the filename that was specified in the
keyword CodeCoverageMergedOutputFile in the build.yaml file.
- job: Code_Coverage
displayName: 'Publish Code Coverage'
dependsOn:
- test_windows_core
- test_linux
pool:
vmImage: 'ubuntu 16.04'
timeoutInMinutes: 0
steps:
- task: DownloadPipelineArtifact@2
displayName: 'Download Pipeline Artifact'
inputs:
buildType: 'current'
artifactName: $(buildArtifactName)
targetPath: '$(Build.SourcesDirectory)/$(buildArtifactName)'
- task: DownloadPipelineArtifact@2 # Downloads the artifact 'CodeCoverageLinux'.
displayName: 'Download Test Artifact Linux'
inputs:
buildType: 'current'
artifactName: 'CodeCoverageLinux'
targetPath: '$(Build.SourcesDirectory)/$(buildFolderName)/$(testResultFolderName)'
- task: DownloadPipelineArtifact@2 # Downloads the artifact 'CodeCoverageWinPS7'.
displayName: 'Download Test Artifact Windows (PS7)'
inputs:
buildType: 'current'
artifactName: 'CodeCoverageWinPS7'
targetPath: '$(Build.SourcesDirectory)/$(buildFolderName)/$(testResultFolderName)'
- task: PowerShell@2 # Merge the code coverage files.
name: merge
displayName: 'Merge Code Coverage files'
inputs:
filePath: './build.ps1'
arguments: '-tasks merge'
pwsh: true
- task: PublishCodeCoverageResults@1 # Must specify the file specified in the `CodeCoverageMergedOutputFile`
displayName: 'Publish Code Coverage to Azure DevOps'
inputs:
codeCoverageTool: 'JaCoCo'
summaryFileLocation: '$(Build.SourcesDirectory)/$(buildFolderName)/$(testResultFolderName)/JaCoCo_coverage.xml'
pathToSources: '$(Build.SourcesDirectory)/$(sourceFolderName)/'
- script: | # Must specify the file specified in the `CodeCoverageMergedOutputFile`
bash <(curl -s https://codecov.io/bash) -f "./$(buildFolderName)/$(testResultFolderName)/JaCoCo_coverage.xml"
displayName: 'Publish Code Coverage to Codecov.io'