Skip to content

Changes to allow tests to be run outside of CI #882

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Feb 16, 2018
Merged

Changes to allow tests to be run outside of CI #882

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
88d19fc
Change logic for test execution to handle running on Mac
JamesWTruher Feb 10, 2018
ad17fe8
Reduce verbosity of build script
JamesWTruher Feb 10, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ For adding/removing resource strings in the `*.resx` files, it is recommended to
#### Tests
Pester-based ScriptAnalyzer Tests are located in `path/to/PSScriptAnalyzer/Tests` folder.

* Ensure Pester 3.4 is installed on the machine
* Ensure Pester 4.1.1 is installed on the machine
* Copy `path/to/PSScriptAnalyzer/out/PSScriptAnalyzer` to a folder in `PSModulePath`
* Go the Tests folder in your local repository
* Run Engine Tests:
Expand Down
15 changes: 6 additions & 9 deletions Tests/Engine/CustomizedRule.tests.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,19 +97,16 @@ Describe "Test importing correct customized rules" {
$customizedRulePath.Count | Should Be 1
}

It "will show the custom rule when given a rule folder path with trailing backslash" {
It "will show the custom rule when given a rule folder path with trailing backslash" -skip:(!$IsWindows){
Copy link
Collaborator

@bergmeister bergmeister Feb 10, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess you are planning to introduce a Travis build for Linux/Mac as well? I think it would be good if you open an issue with a brief description of the high level plan of PR's to come so that we can see the big picture.

Copy link
Contributor Author

@JamesWTruher JamesWTruher Feb 12, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup, that's the plan, I'll work on that issue
the start is here: #886

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Looks good to me.

# needs fixing for linux
if (-not (Test-PSEditionCoreCLRLinux))
{
$customizedRulePath = Get-ScriptAnalyzerRule -CustomizedRulePath $directory/samplerule/ | Where-Object {$_.RuleName -eq $measure}
$customizedRulePath.Count | Should Be 1
}
$customizedRulePath = Get-ScriptAnalyzerRule -CustomizedRulePath $directory/samplerule/ | Where-Object {$_.RuleName -eq $measure}
$customizedRulePath.Count | Should Be 1
}

It "will show the custom rules when given a glob" {
# needs fixing for Linux
$expectedNumRules = 4
if (Test-PSEditionCoreCLRLinux)
if (!$IsWindows)
{
$expectedNumRules = 3
}
Expand All @@ -125,7 +122,7 @@ Describe "Test importing correct customized rules" {
It "will show the custom rules when given glob with recurse switch" {
# needs fixing for Linux
$expectedNumRules = 5
if (Test-PSEditionCoreCLRLinux)
if (!$IsWindows)
{
$expectedNumRules = 4
}
Expand Down Expand Up @@ -165,7 +162,7 @@ Describe "Test importing correct customized rules" {
{
It "will show the custom rule in the results when given a rule folder path with trailing backslash" {
# needs fixing for Linux
if (-not (Test-PSEditionCoreCLRLinux))
if ($IsWindows)
{
$customizedRulePath = Invoke-ScriptAnalyzer $directory\TestScript.ps1 -CustomizedRulePath $directory\samplerule\ | Where-Object {$_.Message -eq $message}
$customizedRulePath.Count | Should Be 1
Expand Down
27 changes: 14 additions & 13 deletions Tests/Engine/InvokeScriptAnalyzer.tests.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -468,33 +468,34 @@ Describe "Test CustomizedRulePath" {

Describe "Test -Fix Switch" {

BeforeEach {
$initialTestScript = Get-Content $directory\TestScriptWithFixableWarnings.ps1 -Raw
BeforeAll {
$scriptName = "TestScriptWithFixableWarnings.ps1"
$testSource = join-path $PSScriptRoot $scriptName
$fixedScript = join-path $PSScriptRoot TestScriptWithFixableWarnings_AfterFix.ps1
$expectedScriptContent = Get-Content $fixedScript -raw
$testScript = Join-Path $TESTDRIVE $scriptName
}

AfterEach {
if ($null -ne $initialTestScript)
{
[System.IO.File]::WriteAllText("$($directory)\TestScriptWithFixableWarnings.ps1", $initialTestScript) # Set-Content -NoNewline was only introduced in PS v5 and would therefore not work in CI
}
BeforeEach {
Copy-Item $testSource $TESTDRIVE
}

It "Fixes warnings" {
# we expect the script to contain warnings
$warningsBeforeFix = Invoke-ScriptAnalyzer $directory\TestScriptWithFixableWarnings.ps1
$warningsBeforeFix = Invoke-ScriptAnalyzer $testScript
$warningsBeforeFix.Count | Should Be 5

# fix the warnings and expect that it should not return the fixed warnings
$warningsWithFixSwitch = Invoke-ScriptAnalyzer $directory\TestScriptWithFixableWarnings.ps1 -Fix
$warningsWithFixSwitch = Invoke-ScriptAnalyzer $testScript -Fix
$warningsWithFixSwitch.Count | Should Be 0

# double check that the warnings are really fixed
$warningsAfterFix = Invoke-ScriptAnalyzer $directory\TestScriptWithFixableWarnings.ps1
$warningsAfterFix = Invoke-ScriptAnalyzer $testScript
$warningsAfterFix.Count | Should Be 0

$expectedScriptContentAfterFix = Get-Content $directory\TestScriptWithFixableWarnings_AfterFix.ps1 -Raw
$actualScriptContentAfterFix = Get-Content $directory\TestScriptWithFixableWarnings.ps1 -Raw
$actualScriptContentAfterFix | Should Be $expectedScriptContentAfterFix
# check content to ensure we have what we expect
$actualScriptContentAfterFix = Get-Content $testScript -Raw
$actualScriptContentAfterFix | Should Be $expectedScriptContent
}
}

Expand Down
179 changes: 89 additions & 90 deletions Tests/Engine/ModuleDependencyHandler.tests.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,51 +1,43 @@
if (!(Get-Module PSScriptAnalyzer) -and !$testingLibraryUsage)
{
Import-Module PSScriptAnalyzer
}

if ($testingLibraryUsage)
{
return
}

$directory = Split-Path -Parent $MyInvocation.MyCommand.Path
$testRootDirectory = Split-Path -Parent $directory
Import-Module (Join-Path $testRootDirectory 'PSScriptAnalyzerTestHelper.psm1')

# needs fixing
# right now we do not support module dependency handing on Linux
if ((Test-PSEditionCoreCLRLinux))
{
return
}

# DSC Module saving is not supported in versions less than PSv5
if (($PSVersionTable.PSVersion -lt [Version]'5.0.0'))
{
return
}

$violationFileName = 'MissingDSCResource.ps1'
$violationFilePath = Join-Path $directory $violationFileName

Describe "Resolve DSC Resource Dependency" {
BeforeAll {
$skipTest = $false
if ( -not $IsWindows -or $testingLibararyUsage -or ($PSversionTable.PSVersion -lt [Version]'5.0.0'))
{
$skipTest = $true
return
}
$SavedPSModulePath = $env:PSModulePath
$violationFileName = 'MissingDSCResource.ps1'
$violationFilePath = Join-Path $directory $violationFileName

Function Test-EnvironmentVariables($oldEnv)
{
$newEnv = Get-Item Env:\* | Sort-Object -Property Key
$newEnv.Count | Should Be $oldEnv.Count
foreach ($index in 1..$newEnv.Count)
$directory = Split-Path -Parent $MyInvocation.MyCommand.Path
$testRootDirectory = Split-Path -Parent $directory
Import-Module (Join-Path $testRootDirectory 'PSScriptAnalyzerTestHelper.psm1')

Function Test-EnvironmentVariables($oldEnv)
{
$newEnv[$index].Key | Should Be $oldEnv[$index].Key
$newEnv[$index].Value | Should Be $oldEnv[$index].Value
$newEnv = Get-Item Env:\* | Sort-Object -Property Key
$newEnv.Count | Should Be $oldEnv.Count
foreach ($index in 1..$newEnv.Count)
{
$newEnv[$index].Key | Should Be $oldEnv[$index].Key
$newEnv[$index].Value | Should Be $oldEnv[$index].Value
}
}
}
AfterAll {
if ( $skipTest ) { return }
$env:PSModulePath = $SavedPSModulePath
}

Context "Module handler class" {
$moduleHandlerType = [Microsoft.Windows.PowerShell.ScriptAnalyzer.Generic.ModuleDependencyHandler]
$oldEnvVars = Get-Item Env:\* | Sort-Object -Property Key
$oldPSModulePath = $env:PSModulePath
It "Sets defaults correctly" {
BeforeAll {
if ( $skipTest ) { return }
$moduleHandlerType = [Microsoft.Windows.PowerShell.ScriptAnalyzer.Generic.ModuleDependencyHandler]
$oldEnvVars = Get-Item Env:\* | Sort-Object -Property Key
$oldPSModulePath = $env:PSModulePath
}
It "Sets defaults correctly" -skip:$skipTest {
$rsp = [runspacefactory]::CreateRunspace()
$rsp.Open()
$depHandler = $moduleHandlerType::new($rsp)
Expand All @@ -69,21 +61,21 @@ Describe "Resolve DSC Resource Dependency" {
$rsp.Dispose()
}

It "Keeps the environment variables unchanged" {
It "Keeps the environment variables unchanged" -skip:$skipTest {
Test-EnvironmentVariables($oldEnvVars)
}

It "Throws if runspace is null" {
It "Throws if runspace is null" -skip:$skipTest {
{$moduleHandlerType::new($null)} | Should Throw
}

It "Throws if runspace is not opened" {
It "Throws if runspace is not opened" -skip:$skipTest {
$rsp = [runspacefactory]::CreateRunspace()
{$moduleHandlerType::new($rsp)} | Should Throw
$rsp.Dispose()
}

It "Extracts 1 module name" {
It "Extracts 1 module name" -skip:$skipTest {
$sb = @"
{Configuration SomeConfiguration
{
Expand All @@ -97,7 +89,7 @@ Describe "Resolve DSC Resource Dependency" {
$resultModuleNames[0] | Should Be 'SomeDscModule1'
}

It "Extracts more than 1 module names" {
It "Extracts more than 1 module names" -skip:$skipTest {
$sb = @"
{Configuration SomeConfiguration
{
Expand All @@ -114,7 +106,7 @@ Describe "Resolve DSC Resource Dependency" {
}


It "Extracts module names when ModuleName parameter is not the first named parameter" {
It "Extracts module names when ModuleName parameter is not the first named parameter" -skip:$skipTest {
$sb = @"
{Configuration SomeConfiguration
{
Expand All @@ -130,64 +122,71 @@ Describe "Resolve DSC Resource Dependency" {
}

Context "Invoke-ScriptAnalyzer without switch" {
It "Has parse errors" {
It "Has parse errors" -skip:$skipTest {
$dr = Invoke-ScriptAnalyzer -Path $violationFilePath -ErrorVariable parseErrors -ErrorAction SilentlyContinue
$parseErrors.Count | Should Be 1
}
}

Context "Invoke-ScriptAnalyzer without switch but with module in temp path" {
$oldEnvVars = Get-Item Env:\* | Sort-Object -Property Key
$moduleName = "MyDscResource"
$modulePath = "$(Split-Path $directory)\Rules\DSCResourceModule\DSCResources\$moduleName"

# Save the current environment variables
$oldLocalAppDataPath = $env:LOCALAPPDATA
$oldTempPath = $env:TEMP
$oldPSModulePath = $env:PSModulePath

# set the environment variables
$tempPath = Join-Path $oldTempPath ([guid]::NewGUID()).ToString()
$newLocalAppDataPath = Join-Path $tempPath "LocalAppData"
$newTempPath = Join-Path $tempPath "Temp"
$env:LOCALAPPDATA = $newLocalAppDataPath
$env:TEMP = $newTempPath

# create the temporary directories
New-Item -Type Directory -Path $newLocalAppDataPath
New-Item -Type Directory -Path $newTempPath

# create and dispose module dependency handler object
# to setup the temporary module
$rsp = [runspacefactory]::CreateRunspace()
$rsp.Open()
$depHandler = [Microsoft.Windows.PowerShell.ScriptAnalyzer.Generic.ModuleDependencyHandler]::new($rsp)
$pssaAppDataPath = $depHandler.PSSAAppDataPath
$tempModulePath = $depHandler.TempModulePath
$rsp.Dispose()
$depHandler.Dispose()

# copy myresource module to the temporary location
# we could let the module dependency handler download it from psgallery
Copy-Item -Recurse -Path $modulePath -Destination $tempModulePath

It "Doesn't have parse errors" {
BeforeAll {
if ( $skipTest ) { return }
$oldEnvVars = Get-Item Env:\* | Sort-Object -Property Key
$moduleName = "MyDscResource"
$modulePath = "$(Split-Path $directory)\Rules\DSCResourceModule\DSCResources\$moduleName"

# Save the current environment variables
$oldLocalAppDataPath = $env:LOCALAPPDATA
$oldTempPath = $env:TEMP
$oldPSModulePath = $env:PSModulePath

# set the environment variables
$tempPath = Join-Path $oldTempPath ([guid]::NewGUID()).ToString()
$newLocalAppDataPath = Join-Path $tempPath "LocalAppData"
$newTempPath = Join-Path $tempPath "Temp"
$env:LOCALAPPDATA = $newLocalAppDataPath
$env:TEMP = $newTempPath

# create the temporary directories
New-Item -Type Directory -Path $newLocalAppDataPath
New-Item -Type Directory -Path $newTempPath

# create and dispose module dependency handler object
# to setup the temporary module
$rsp = [runspacefactory]::CreateRunspace()
$rsp.Open()
$depHandler = [Microsoft.Windows.PowerShell.ScriptAnalyzer.Generic.ModuleDependencyHandler]::new($rsp)
$pssaAppDataPath = $depHandler.PSSAAppDataPath
$tempModulePath = $depHandler.TempModulePath
$rsp.Dispose()
$depHandler.Dispose()

# copy myresource module to the temporary location
# we could let the module dependency handler download it from psgallery
Copy-Item -Recurse -Path $modulePath -Destination $tempModulePath
}

AfterAll {
if ( $skipTest ) { return }
#restore environment variables and clean up temporary location
$env:LOCALAPPDATA = $oldLocalAppDataPath
$env:TEMP = $oldTempPath
Remove-Item -Recurse -Path $tempModulePath -Force
Remove-Item -Recurse -Path $tempPath -Force
}

It "Doesn't have parse errors" -skip:$skipTest {
# invoke script analyzer
$dr = Invoke-ScriptAnalyzer -Path $violationFilePath -ErrorVariable parseErrors -ErrorAction SilentlyContinue
$dr.Count | Should Be 0
}

It "Keeps PSModulePath unchanged before and after invocation" {
It "Keeps PSModulePath unchanged before and after invocation" -skip:$skipTest {
$dr = Invoke-ScriptAnalyzer -Path $violationFilePath -ErrorVariable parseErrors -ErrorAction SilentlyContinue
$env:PSModulePath | Should Be $oldPSModulePath
}
#restore environment variables and clean up temporary location
$env:LOCALAPPDATA = $oldLocalAppDataPath
$env:TEMP = $oldTempPath
Remove-Item -Recurse -Path $tempModulePath -Force
Remove-Item -Recurse -Path $tempPath -Force

It "Keeps the environment variables unchanged" {
It "Keeps the environment variables unchanged" -skip:$skipTest {
Test-EnvironmentVariables($oldEnvVars)
}
}
Expand Down
6 changes: 3 additions & 3 deletions Tests/Engine/ModuleHelp.Tests.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ Enter the version of the module to test. This parameter is optional. If you
omit it, the test runs on the latest version of the module in $env:PSModulePath.

.EXAMPLE
.\Module.Help.Tests.ps1 -ModuleName Pester -RequiredVersion 3.4.0
This command runs the tests on the commands in Pester 3.4.0.
.\Module.Help.Tests.ps1 -ModuleName Pester -RequiredVersion 4.1.1
This command runs the tests on the commands in Pester 4.1.1.

.EXAMPLE
.\Module.Help.Tests.ps1 -ModuleName Pester
Expand Down Expand Up @@ -63,7 +63,7 @@ Param
$RequiredVersion
)

# #Requires -Module @{ModuleName = 'Pester'; ModuleVersion = '3.4.0'}
# #Requires -Module @{ModuleName = 'Pester'; ModuleVersion = '4.1.1'}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a technical reason to upgrade to v4 as part of this PR? I think it might be better to do this as a separate PR?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the test fixes are sort of entwined, by doing them all at once I won't need to change the tests again.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I see. Then we also need to update the documented version number of Pester in the test section here of the developer guide.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

quite right - fixed


<#
.SYNOPSIS
Expand Down
4 changes: 2 additions & 2 deletions Tests/Engine/RuleSuppression.tests.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ function SuppressPwdParam()
}

Context "Rule suppression within DSC Configuration definition" {
It "Suppresses rule" -skip:((Test-PSEditionCoreCLRLinux) -or ($PSVersionTable.PSVersion -lt [Version]'5.0.0')) {
It "Suppresses rule" -skip:((!$IsWindows) -or ($PSVersionTable.PSVersion.Major -lt 5)) {
$suppressedRule = Invoke-ScriptAnalyzer -ScriptDefinition $ruleSuppressionInConfiguration -SuppressedOnly
$suppressedRule.Count | Should Be 1
}
Expand Down Expand Up @@ -215,4 +215,4 @@ Describe "RuleSuppressionWithScope" {
$suppressed.Count | Should Be 1
}
}
}
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a change of the file's encoding?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it doesn't look like it. it looks like there was previously no newline at the end of the file.

Loading