[test_runner] test.py should presumably support errors in a non-entry-point library

[eernstg]

I'd expect the test runner to recognize an error expectation in a library which is not the entry point, and consider an actual occurrence of such an error as a successful test run. For example:

// Store this as library 'tests/language/scratch_test.dart'.
import 'scratch_lib.dart';
main() => C();

// Store this as library 'tests/language/scratch_lib.dart'.
class C extends C {}
//    ^
// [analyzer] unspecified
// [cfe] unspecified

Running a fresh dartanalyzer from commit 65fab23, we get the following error message:

ERROR|COMPILE_TIME_ERROR|RECURSIVE_INTERFACE_INHERITANCE_EXTENDS|/usr/local/google/home/eernst/devel/dart/work/sdk/tests/language/scratch_lib.dart|1|7|1|'C' can't extend itself.

This error message should be matched by the expectation comment in scratch_lib.dart, but we still get the following outcome:

> cd $SDK_HOME
> python tools/test.py -n analyzer-asserts-strong-linux --verbose language/scratch_test
Test configuration:
    analyzer-asserts-strong-linux(architecture: x64, compiler: dart2analyzer, mode: release, runtime: none, system: linux, nnbd: strong, enable-asserts, use-sdk)
Suites tested: language
Running "dart2analyzer" command: DART_CONFIGURATION=ReleaseX64 out/ReleaseX64/dart-sdk/bin/dartanalyzer --ignore-unrecognized-flags --packages=/usr/local/google/home/eernst/devel/dart/work/sdk/.packages --format=machine --no-hints /usr/local/google/home/eernst/devel/dart/work/sdk/tests/language/scratch_test.dart

FAILED: dart2analyzer-none release_x64 language/scratch_test
Expected: Pass
Actual: CompileTimeError

--- Command "dart2analyzer" (took 02.000283s):
DART_CONFIGURATION=ReleaseX64 out/ReleaseX64/dart-sdk/bin/dartanalyzer --ignore-unrecognized-flags --packages=/usr/local/google/home/eernst/devel/dart/work/sdk/.packages --format=machine --no-hints /usr/local/google/home/eernst/devel/dart/work/sdk/tests/language/scratch_test.dart

unexpected analysis errors:
In tests/language/scratch_lib.dart:
- Line 1, column 7: COMPILE_TIME_ERROR.RECURSIVE_INTERFACE_INHERITANCE_EXTENDS
  'C' can't extend itself.

--- Re-run this test:
python tools/test.py -n analyzer-asserts-strong-linux language/scratch_test
Done dart2analyzer-none release_x64 language/scratch_test: fail

=== 0 tests passed, 1 failed ===

In other words, the test runner does not seem to expect that an error expectation comment can occur in any other library of a test than the entry point.

Motivation This can be a problem in practice: Some tests are intended to handle mixed-version programs (some libraries opt in null safety, others opt it out), and they generally need to have an entry point which is opted out (in order to avoid diagnostic messages about reverse imports, optedIn --> outedOut). Next, some errors occur in opted-in code, so they can't be in the entry point.

It is possible that the error expectation comments are supposed to work also in non-entry-point libraries of a test, in which case this issue should be labeled as type-bug. However, it could also be considered a new and possibly useful behavior, in which case it should be labeled as type-enhancement. So I haven't added any labels about this dimension.

[eernstg]

@munificent, what do you think?

[munificent]

Yeah, this is reasonable. The static error test stuff doesn't currently support this because we didn't have a need for it in the tests I'd seen but it makes sense to add support for this.

[sgrekhov]

I hit this as well.

// lib.dart
library [Libraries_and_Scripts_A03_t10_lib];
//      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
// [analyzer] unspecified
// [cfe] unspecified

// test.dart
import "lib.dart";

main() {
}

How to expect static error here? I want to check here that there is a compile time error if library name is wrong. But there are no erroneous lines in a test.dart and expectations in a lib.dart don't work

[eernstg]

@munificent, I think it's very likely that the tests that need to have expected errors outside the entry point could be written in such a way that they only have those errors in one library. Would it be easier to adjust the test runner such that it could handle this situation, e.g., based on a directive in the entry point?:

// ----- 'my_test.dart'
// ExpectedErrorsIn='my_test_part.dart'

part 'my_test_part.dart';
void main() {}

// ----- my_test_part.dart'

part of 'some_other_library.dart';
//      ^^^^^^^^^^^^^^^^^^^^^^^^^
// [analyzer] Some error message about the wrong part-of URI.

This might also help avoiding a performance problem: If the test runner always needs to scan through every reachable library to see if it contains any error expectations then this might slow down test runs quite a lot.

[scheglov]

FWIW, this issue accounts for 90 out of 269 tests that fail on analyzer-asserts-strong-linux.