Manual testing of Release 2022.10

[alisevych]

Initial set-up

Check that the IntelliJ Idea UTBot plugin can be successfully installed on IDEA 2022.1.4

• Plugin from unit-test-bot/rc2992022 build with latest changes
• Download plugin
• Check downloaded zip-file size < 100 MB
• Open IntelliJ IDE 2022.1.4
• Remove previously installed UTBot plugin
• Clone or reuse UTBot project (https://github.com/UnitTestBot/UTBotJava.git)
• Open the project in the IDE
• Install the downloaded plugin

Check that the IntelliJ Idea UTBot plugin can be successfully installed on IDEA 2022.2.2

• Open IntelliJ IDE 2022.2.2
• Remove previously installed UTBot plugin
• Clone or reuse UTBot project (https://github.com/UnitTestBot/UTBotJava.git)
• Open the project in the IDE
• Install the downloaded plugin

Go through manual scenarios on IDEA 2022.1.4

Manual scenario #1

• Use default plugin settings
• Open the utbot-sample/src/main/java/org/utbot/examples/algorithms/ArraysQuickSort.java file
• Generate tests for the class
• Remove results
• Generate and Run test for a method

Manual scenario #2

• Use default plugin settings
• Open the utbot-sample/src/main/java/org/utbot/examples/mock/CommonMocksExample.java file
• Generate tests with different Mocking options combinations

Go through manual scenarios on IDEA 2022.2.2

Manual scenario #1

• Use default plugin settings
• Open the utbot-sample/src/main/java/org/utbot/examples/algorithms/ArraysQuickSort.java file
• Generate tests for the class
• Remove results
• Generate and Run test for a method

Manual scenario #2

• Use default plugin settings
• Open the utbot-sample/src/main/java/org/utbot/examples/mock/CommonMocksExample.java file
• Generate tests with different Mocking options combinations

@alisevych

Check following in IDEA 2022.1.4

Manual scenario #3

• Create a new Gradle project with JDK 8
• Add a simple java file to test
• Generate a test in the existing test root

Manual scenario #4

• Create a new Maven project with JDK 8
• Add a simple java file to test
• Generate a test with existing test sources root

Manual scenario #5

• Create a new Idea project with JDK 11
• Add a simple java file to test
• Generate tests for several classes in a new test sources root

@tyuldashev

Check following in IDEA 2022.2.2

Manual scenario #3

• Create a new Gradle project with JDK 8
• Add a simple java file to test
• Generate a test in the existing test root

Manual scenario #4

• Create a new Maven project with JDK 8
• Add a simple java file to test
• Generate a test with existing test sources root

Manual scenario #5

• Create a new Idea project with JDK 11
• Add a simple java file to test
• Generate tests for several classes in a new test sources root

Regression testing

@alisevych

IDEA 2022.1.4

• Maven with JDK 11 : make following checks
• Maven with JDK 8 : make following checks
• Test frameworks JUnit4/ JUnit5/TestNG
• - with Parametrization on / off
• - with mocks on /off
• - UtUtils class (with mocks off / on) - check it's regenerated when mocks are on for the first time
• String Java 8 - with JDK 8
• String Java 9 - 11 - with JDK 11

Project with JDK 11:

• timeout changes - 1/10/60 sec
• cancellation
• regions: Symbolic first, Fuzzing, error suite (NPE), explicitly thrown exceptions, caught exceptions (successful)
• exceptions - check regions for exceptions
• context menu / shortcut
• call from method / class / package - check Members list
• math
• make.symbolic
• inner
• collections
• Generics - with Fuzzing
• Sandbox - smoke

@tyuldashev

IDEA 2022.2.2

• Gradle project with JDK 8 : make following checks
• Gradle project with JDK 11 : make following checks
• Test frameworks JUnit4/ JUnit5/TestNG
• - with Parametrization on / off
• - with mocks on /off
• - UtUtils class (with mocks off / on) - check it's regenerated when mocks are on for the first time
• String Java 8 - with JDK 8
• String Java 9 - 11 - with JDK 11

Project with JDK 11:

• timeout changes - 1/10/60 sec
• cancellation
• regions: Symbolic first, Fuzzing, error suite (NPE), explicitly thrown exceptions, caught exceptions (successful)
• exceptions - check regions for exceptions
• context menu / shortcut
• call from method / class / package - check Members list
• Date - check
• Arrays
• annotations
• Generics - with Fuzzing
• Sandbox - smoke

[alisevych]

reproducing #1028 - for Recursion
reproducing #1054 - in IDEA 2022.1.4 (JBR 11), 2022.2.2 (JBR 17)
reproducing #980
reproducing #1064
reproducing #747 - for utbot-sample/src/main/java/org/utbot/examples/collections/CustomerExamples.java

[alisevych]

new bug with Recursion - empty test class is generated, StackOverflow in Concrete executor log
(not always reproducing)
reproducing with Cycles
#1130

[alisevych]

new bug with MockRandomExample - Fuzzing is generating false-successful test
#1135

[alisevych]

new bug? - StringExamples hanging, cancellation doesn't help
Illegal reflective access in concrete executor logs, error test for IntPair:
/* cannot get resident body for phantom method : <java.lang.Object: void
()> */
#1141

[alisevych]

question - conditions in parametrized tests : if (expectedResult == null) {} else {}
better generate 2 separate tests without conditions.

created #1140

[alisevych]

question - EXPLICITLY_THROWN_UNCHECKED_EXCEPTIONS - String.repeat(-1) throws [java.lang.IllegalArgumentException: count is negative: -1]
is it correct, that explicitly?
#1133

[tyuldashev]

Reproducing:

• Hanging test timeout is ignored by Symbolic execution #1095
• Project version of 'utbot-api' used instead of plugin one #1099

[alisevych]

new bug: java.lang.reflect.InvocationTargetException
для ClassWithComplicatedMethods.sqrt2()
with Fuzzing
#1117