Skip to content

Improve sandbox-relative description in generated tests #782

@Markoutte

Description

@Markoutte
Collaborator

Description

After #625 was implemented methods that try to run some unsafe code now generates into tests like this:

@Disabled("Disabled due to sandbox")
public void testPropertyWithBlankString() {
        SecurityCheck securityCheck = new SecurityCheck();
        
        /* This test fails because method [com.company.security.SecurityCheck.property] produces [java.security.AccessControlException: access denied ("java.util.PropertyPermission" "   " "read")]
            java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
            java.security.AccessController.checkPermission(AccessController.java:886)
            java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
            java.lang.SecurityManager.checkPropertyAccess(SecurityManager.java:1294)
            java.lang.System.getProperty(System.java:719)
            com.company.security.SecurityCheck.property(SecurityCheck.java:32) */
    }

We have several problems with this view:

  1. User have no information how to change settings of security manager
  2. TestNG has no @Disabled annotation
  3. Is it good idea to print stacktrace?

Expected behavior

Generated test should be improved. We can add little doc or even hyperlink to add permissions into correct file.

Activity

self-assigned this
on Aug 24, 2022
moved this to Todo in UTBot Javaon Aug 24, 2022
alisevych

alisevych commented on Aug 24, 2022

@alisevych
Member

@Markoutte @EgorkaKulikov
Could you please also consider the following:

  1. Disabling tests is made in a bit different way in different test frameworks:
    JUnit4 has only @Ignore("<comment>")
    JUnit5 has @Disabled("<comment>")
    TestNG has possibility to disable tests inside @Test( enabled=false ), there can be also description parameter inside, that Summaries is using. So need to render it once and correctly.

  2. How these tests will be rendered when parameterization is on?

alisevych

alisevych commented on Aug 25, 2022

@alisevych
Member

@Markoutte @EgorkaKulikov
Tested cases from previous comment - tests are rendered correctly.

alisevych

alisevych commented on Sep 26, 2022

@alisevych
Member

"This test fails.." is not a correct note. The test would pass if enabled.
"The test was not generated..." would be more clear. Or "The test generation stopped on unpermitted operation..."
Also it would be great to suggest the user to allow UTBot to generate and run the tests for these unsafe operations - @Markoutte 's idea.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

ctg-enhancementNew feature, improvement or change request

Type

No type

Projects

Status

Todo

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @Markoutte@alisevych

      Issue actions

        Improve sandbox-relative description in generated tests · Issue #782 · UnitTestBot/UTBotJava