Skip to content

Conversation

denscollo
Copy link
Contributor

Fixes #4545

Description

The functional tests currently run and end as Inconclusive, which hides the actual state of the tests. We need to determine if the failures are test issues and if they are, fix them.

The inconclusive result of Facebook's and Webex's tests are a product of required environment variables being un-reachable, which skips the actual functional tests. Our tests ran successfully whenever all environment variables were set up correctly, and we could only replicate the issue's reported behavior when leaving variables null purposefully.

It should be noted that the current inconclusive response is not an issue of the test's implementation. Changing the inconclusive assertions to failures would lead to false negatives, since the tests did not fail, but were unable to be run.

Our proposed solution is to integrate a required variable validating task to run prior to all other tasks in the pipeline. This keeps the tests intact and saves waiting for the pipeline to run until failure due to missing, empty or white-spaced variables, making those occurrences easier to catch and correct.

We validate for required variables mannually on each case, leaving room for custom behavior. It should be noted that we also left the FacebookTestBotFacebookVerifyToken variable on botbuilder-dotnet-ci-facebook-test.yml unvalidated as it was detected that it was not required for the test to run.

Specific Changes

Testing

The following image showcases the new task in action. First we deleted required variables, then the task outputs which variables were missing.

image

We also checked that both Facebook's and Webex's tests were running correctly, which we can corroborate with the following images.

  • Facebook:
    image

  • Webex:
    image

Added variable validation tasks and further steps skip conditions to webex's and facebook's ci ymls
@denscollo denscollo requested a review from a team as a code owner September 4, 2020 12:35
@mrivera-ms
Copy link
Contributor

@denscollo , please fix the conflicts so we can merge this fix. Thank you.

@mrivera-ms mrivera-ms merged commit ec57023 into microsoft:main Sep 14, 2020
@iscaltritti iscaltritti deleted the southworks/add/adapter-variables-validation branch December 11, 2020 21:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Facebook and Webex functional tests fail when not skipped
3 participants