Skip to content
This repository was archived by the owner on Nov 19, 2024. It is now read-only.

Added Static and SemVer tests docs #7951

Merged
merged 18 commits into from
Oct 12, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions src/_data/main-nav.yml
Original file line number Diff line number Diff line change
Expand Up @@ -179,8 +179,6 @@
- label: Web API Functional Testing
url: /get-started/web-api-functional-testing.html

-

- label: Functional Areas
children:

Expand Down
14 changes: 13 additions & 1 deletion src/_data/toc/testing.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,4 +62,16 @@ pages:
children:

- label: '@dataProvider'
url: /test/unit/annotations/data-provider.html
url: /test/unit/annotations/data-provider.html

- label: Static Testing
children:

- label: Running Static Tests
url: /test/static/static_test_execution.html

- label: Semantic Version Checker
children:

- label: Running SVC tool
url: /test/svc/semver_test_execution.html
55 changes: 55 additions & 0 deletions src/guides/v2.3/test/static/static_test_execution.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
---
group: testing
title: Running Static Tests
functional_areas:
- Testing
- test
---

Executing Magento 2 static tests is straightforward. They can be executed in several ways.

### Running static tests on all files

To run static tests on all files, navigate to the Magento base directory and execute the following command:

```bash
bin/magento dev:test:run static
Copy link
Contributor

@hostep hostep Oct 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gabrieldagama: out of curiosity, are these the exact same tests that are being ran when you submit a PR on github?
Unless something has changed recently, that's not the case. It seems like there are a bunch of extra static tests which are being executed on github.
I'm looking for easy to comprehend information about how to execute all these static tests on your local machine, it will save so much waiting time when working on a PR if we have this information.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @hostep, this is a valid point, thanks for bringing it up! Yes, they are the same tests, but they run a little bit differently, on our automated tests, the Static Tests are split in their separate test suite respectively (Less, HTML, PHP, Integrity...) and when you run locally, it runs all the suites at once. We should get the same results locally.

But there are somethings that I've noticed:

  • Some of the tests are run based on changed files (code style, for example), so if you commit your changes before running the tests you may not get the errors.
  • We do have some differences in EE and B2B versions, so if you don't have those installed you may get different results.

Maybe we can add this information to the docs as well, what do you think?

Hope this clarifies a little bit your question. If you have some PR that we can look at to see the differences it would be great!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok thanks for the info!

Ah yes, I seem to remember that it was almost always some extra tests in B2B that failed. Unfortunately we are only using Open Source so I have no access to those extra tests.

What is the reason for the static tests to be different on EE & B2B projects btw? These are static tests, in my opinion they should be 100% the same between the different type of Magento projects. I understand there is a difference for integration & functional tests, but I don't understand why there is a difference for static tests?
Would be great if the differences in static tests could get aligned so they are the same over all types of Magento projects, no?

Copy link
Contributor Author

@gabrieldagama gabrieldagama Oct 6, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with you, from what I can see the PHP check are exactly the same, it just changes some files that are ignored or not. But, there are some differences in other static tests, like Integrity and Legacy tests. I'm not fully aware of exactly the reasons and what are the differences, but I may be able to check internally.

But yeah, from my point of view, ideally we would have a defined test strategy that is publicly available and applied to all versions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also seem to remember vaguely that php mess detector static test ran on B2B and not on OS, but I might be mistaken

```

### Running PHP static tests on a subset of files

To run the static tests on a subset of files, create a new testsuite for phpunit:

1. From the Magento base directory navigate to `dev/tests/static/testsuite/Magento/Test`

1. Create a copy of the `Php` folder on the same directory and rename it to `Local`

1. Navigate to `dev/tests/static/testsuite/Magento/Test/Local/_files/whitelist` and open `common.txt`

1. Replace the contents with the folder of the files that you want to test. For example:

```text
# Format: <componentType=module|library|theme|language|*> <componentName> <globPattern> or simply <globPattern>
app/code/Magento/CatalogSearch/Model/Search
```

1. Create a copy of the `dev/tests/static/phpunit.xml.dist` file and call it `phpunit.xml`:

1. Add a new testsuite to the `dev/tests/static/phpunit.xml` file inside the `<testsuites>` node:

```xml
<testsuites>
...
<testsuite name="Local Test Suite">
<file>testsuite/Magento/Test/Local/LiveCodeTest.php</file>
</testsuite>
</testsuites>
```

1. Navigate to the Magento base directory and run:

```bash
./vendor/bin/phpunit --testsuite="Local Test Suite" -c dev/tests/static/phpunit.xml.dist
```

As a result of this process, you can run PHP static tests on a subset of files. It is also possible to run other types of static tests by following the same process with other testsuites.
36 changes: 36 additions & 0 deletions src/guides/v2.3/test/svc/semver_test_execution.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
---
group: testing
title: Running Semantic Version Checker
functional_areas:
- Testing
- test
---

Executing the Magento 2 Semantic Version Checker is a simple process, but it requires an external tool.

### Running Semantic Version Checker on local changes

To run the Semantic Version Checker on local changes:

1. Clone the `magento-semver` tool to your local machine:

```bash
git clone [email protected]:magento/magento-semver.git
```

1. Navigate to the cloned repository and run `composer install` to install the project dependencies:

```bash
cd magento-semver && composer install
```

1. To run the Semantic Version Checker, we need one folder with the feature changes and another folder with mainline code (without changes), so you may need to clone the Magento repository to a separate folder to perform the comparison.

1. Navigate the `magento-semver` folder and run the Semantic Version Checker compare command:

```bash
bin/svc compare ../magento2-mainline ../magento2
```

The first parameter is the mainline code without any changes, and the second parameter is the path to the folder with your changes.
The results of the Semantic Version Checker are outputted to the console.
1 change: 1 addition & 0 deletions src/guides/v2.4/test/static/static_test_execution.md
1 change: 1 addition & 0 deletions src/guides/v2.4/test/svc/semver_test_execution.md