[Utils] Add new --update-tests flag to llvm-lit #108425
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hnrklssn
requested review from
nikic,
nhaehnle,
jroelofs,
arichardson,
jdenny-ornl,
jh7370 and
Harini0924
llvmbot
added
clang
|
@llvm/pr-subscribers-testing-tools Author: Henrik G. Olsson (hnrklssn) ChangesThis adds a flag to lit for detecting and updating failing tests when possible to do so automatically. The flag uses a plugin architecture where config files can add additional auto-updaters for the types of tests in the test suite. When a test fails with Initially adds support for UpdateVerifyTests and UpdateTestChecks. The flag is currently only implemented for lit's internal shell, so Builds on work in #97369 Patch is 47.68 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/108425.diff 50 Files Affected:
diff --git a/clang/test/Sema/lit.local.cfg b/clang/test/Sema/lit.local.cfg
index baf1b39ef238cd..b385fd92098a8e 100644
--- a/clang/test/Sema/lit.local.cfg
+++ b/clang/test/Sema/lit.local.cfg
@@ -2,3 +2,15 @@ config.substitutions = list(config.substitutions)
config.substitutions.insert(
0, (r"%clang\b", """*** Do not use the driver in Sema tests. ***""")
)
+
+if lit_config.update_tests:
+ import sys
+ import os
+
+ curdir = os.path.dirname(os.path.realpath(__file__))
+ testdir = os.path.dirname(curdir)
+ clangdir = os.path.dirname(testdir)
+ utilspath = os.path.join(clangdir, "utils")
+ sys.path.append(utilspath)
+ from UpdateVerifyTests.litplugin import verify_test_updater
+ lit_config.test_updaters.append(verify_test_updater)
diff --git a/clang/test/lit.cfg.py b/clang/test/lit.cfg.py
index 92a3361ce672e2..c965d3167fe8eb 100644
--- a/clang/test/lit.cfg.py
+++ b/clang/test/lit.cfg.py
@@ -362,3 +362,12 @@ def calculate_arch_features(arch_string):
# possibly be present in system and user configuration files, so disable
# default configs for the test runs.
config.environment["CLANG_NO_DEFAULT_CONFIG"] = "1"
+
+if lit_config.update_tests:
+ import sys
+ import os
+
+ utilspath = os.path.join(config.llvm_src_root, "utils")
+ sys.path.append(utilspath)
+ from update_any_test_checks import utc_lit_plugin
+ lit_config.test_updaters.append(utc_lit_plugin)
diff --git a/clang/test/utils/update-verify-tests/Inputs/duplicate-diag.c b/clang/test/utils/update-verify-tests/Inputs/duplicate-diag.c
new file mode 100644
index 00000000000000..d4a92eb4a7874a
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/duplicate-diag.c
@@ -0,0 +1,10 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ // expected-error@+1{{use of undeclared identifier 'a'}}
+ a = 2; a = 2;
+ b = 2; b = 2;
+ // expected-error@+1 3{{use of undeclared identifier 'c'}}
+ c = 2; c = 2;
+ // expected-error 2{{asdf}}
+}
diff --git a/clang/test/utils/update-verify-tests/Inputs/duplicate-diag.c.expected b/clang/test/utils/update-verify-tests/Inputs/duplicate-diag.c.expected
new file mode 100644
index 00000000000000..5fd72709e487f5
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/duplicate-diag.c.expected
@@ -0,0 +1,10 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ // expected-error@+1 2{{use of undeclared identifier 'a'}}
+ a = 2; a = 2;
+ // expected-error@+1 2{{use of undeclared identifier 'b'}}
+ b = 2; b = 2;
+ // expected-error@+1 2{{use of undeclared identifier 'c'}}
+ c = 2; c = 2;
+}
diff --git a/clang/test/utils/update-verify-tests/Inputs/infer-indentation.c b/clang/test/utils/update-verify-tests/Inputs/infer-indentation.c
new file mode 100644
index 00000000000000..9a58244d92e066
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/infer-indentation.c
@@ -0,0 +1,10 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ // expected-error@+1 2 {{use of undeclared identifier 'a'}}
+ a = 2; a = 2; b = 2; b = 2; c = 2;
+ // expected-error@+1 2 {{asdf}}
+ d = 2;
+ e = 2; f = 2; // expected-error 2 {{use of undeclared identifier 'e'}}
+}
+
diff --git a/clang/test/utils/update-verify-tests/Inputs/infer-indentation.c.expected b/clang/test/utils/update-verify-tests/Inputs/infer-indentation.c.expected
new file mode 100644
index 00000000000000..127efb003b1883
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/infer-indentation.c.expected
@@ -0,0 +1,13 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ // expected-error@+3 {{use of undeclared identifier 'c'}}
+ // expected-error@+2 2 {{use of undeclared identifier 'b'}}
+ // expected-error@+1 2 {{use of undeclared identifier 'a'}}
+ a = 2; a = 2; b = 2; b = 2; c = 2;
+ // expected-error@+1 {{use of undeclared identifier 'd'}}
+ d = 2;
+ // expected-error@+1 {{use of undeclared identifier 'f'}}
+ e = 2; f = 2; // expected-error {{use of undeclared identifier 'e'}}
+}
+
diff --git a/clang/test/utils/update-verify-tests/Inputs/leave-existing-diags.c b/clang/test/utils/update-verify-tests/Inputs/leave-existing-diags.c
new file mode 100644
index 00000000000000..075d6c80fd4c36
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/leave-existing-diags.c
@@ -0,0 +1,13 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ a = 2;
+ // expected-error@-1{{use of undeclared identifier 'a'}}
+ b = 2;// expected-error{{use of undeclared identifier 'b'}}
+ c = 2;
+ // expected-error@7{{use of undeclared identifier 'c'}}
+ d = 2; // expected-error-re{{use of {{.*}} identifier 'd'}}
+
+ e = 2; // error to trigger mismatch
+}
+
diff --git a/clang/test/utils/update-verify-tests/Inputs/leave-existing-diags.c.expected b/clang/test/utils/update-verify-tests/Inputs/leave-existing-diags.c.expected
new file mode 100644
index 00000000000000..778ebd4ff4c7b9
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/leave-existing-diags.c.expected
@@ -0,0 +1,14 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ a = 2;
+ // expected-error@-1{{use of undeclared identifier 'a'}}
+ b = 2;// expected-error{{use of undeclared identifier 'b'}}
+ c = 2;
+ // expected-error@7{{use of undeclared identifier 'c'}}
+ d = 2; // expected-error-re{{use of {{.*}} identifier 'd'}}
+
+ // expected-error@+1{{use of undeclared identifier 'e'}}
+ e = 2; // error to trigger mismatch
+}
+
diff --git a/clang/test/utils/update-verify-tests/Inputs/multiple-errors.c b/clang/test/utils/update-verify-tests/Inputs/multiple-errors.c
new file mode 100644
index 00000000000000..d3fa93ec0e8221
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/multiple-errors.c
@@ -0,0 +1,8 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ a = 2;
+ b = 2;
+
+ c = 2;
+}
diff --git a/clang/test/utils/update-verify-tests/Inputs/multiple-errors.c.expected b/clang/test/utils/update-verify-tests/Inputs/multiple-errors.c.expected
new file mode 100644
index 00000000000000..aa0cf22c8e8a7b
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/multiple-errors.c.expected
@@ -0,0 +1,11 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ // expected-error@+1{{use of undeclared identifier 'a'}}
+ a = 2;
+ // expected-error@+1{{use of undeclared identifier 'b'}}
+ b = 2;
+
+ // expected-error@+1{{use of undeclared identifier 'c'}}
+ c = 2;
+}
diff --git a/clang/test/utils/update-verify-tests/Inputs/multiple-missing-errors-same-line.c b/clang/test/utils/update-verify-tests/Inputs/multiple-missing-errors-same-line.c
new file mode 100644
index 00000000000000..9c5104e693ae57
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/multiple-missing-errors-same-line.c
@@ -0,0 +1,10 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ a = 2; b = 2; c = 2;
+}
+
+void bar() {
+ x = 2; y = 2; z = 2;
+ // expected-error@-1{{use of undeclared identifier 'x'}}
+}
diff --git a/clang/test/utils/update-verify-tests/Inputs/multiple-missing-errors-same-line.c.expected b/clang/test/utils/update-verify-tests/Inputs/multiple-missing-errors-same-line.c.expected
new file mode 100644
index 00000000000000..01076f76318997
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/multiple-missing-errors-same-line.c.expected
@@ -0,0 +1,15 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ // expected-error@+3{{use of undeclared identifier 'c'}}
+ // expected-error@+2{{use of undeclared identifier 'b'}}
+ // expected-error@+1{{use of undeclared identifier 'a'}}
+ a = 2; b = 2; c = 2;
+}
+
+void bar() {
+ x = 2; y = 2; z = 2;
+ // expected-error@-1{{use of undeclared identifier 'x'}}
+ // expected-error@-2{{use of undeclared identifier 'y'}}
+ // expected-error@-3{{use of undeclared identifier 'z'}}
+}
diff --git a/clang/test/utils/update-verify-tests/Inputs/no-checks.c b/clang/test/utils/update-verify-tests/Inputs/no-checks.c
new file mode 100644
index 00000000000000..019698122c3f2b
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/no-checks.c
@@ -0,0 +1,5 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ bar = 2;
+}
diff --git a/clang/test/utils/update-verify-tests/Inputs/no-checks.c.expected b/clang/test/utils/update-verify-tests/Inputs/no-checks.c.expected
new file mode 100644
index 00000000000000..42c98d4f77b6ce
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/no-checks.c.expected
@@ -0,0 +1,6 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ // expected-error@+1{{use of undeclared identifier 'bar'}}
+ bar = 2;
+}
diff --git a/clang/test/utils/update-verify-tests/Inputs/no-diags.c b/clang/test/utils/update-verify-tests/Inputs/no-diags.c
new file mode 100644
index 00000000000000..fddc2ad661fd2d
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/no-diags.c
@@ -0,0 +1,7 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ // expected-error@+1{{asdf}}
+ int a = 2;
+}
+
diff --git a/clang/test/utils/update-verify-tests/Inputs/no-diags.c.expected b/clang/test/utils/update-verify-tests/Inputs/no-diags.c.expected
new file mode 100644
index 00000000000000..acb02badc5655a
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/no-diags.c.expected
@@ -0,0 +1,7 @@
+// expected-no-diagnostics
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ int a = 2;
+}
+
diff --git a/clang/test/utils/update-verify-tests/Inputs/no-expected-diags.c b/clang/test/utils/update-verify-tests/Inputs/no-expected-diags.c
new file mode 100644
index 00000000000000..e02b63dcd7a21b
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/no-expected-diags.c
@@ -0,0 +1,6 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+// expected-no-diagnostics
+void foo() {
+ a = 2;
+}
diff --git a/clang/test/utils/update-verify-tests/Inputs/no-expected-diags.c.expected b/clang/test/utils/update-verify-tests/Inputs/no-expected-diags.c.expected
new file mode 100644
index 00000000000000..7fd85f2e064e28
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/no-expected-diags.c.expected
@@ -0,0 +1,6 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ // expected-error@+1{{use of undeclared identifier 'a'}}
+ a = 2;
+}
diff --git a/clang/test/utils/update-verify-tests/Inputs/non-default-prefix.c b/clang/test/utils/update-verify-tests/Inputs/non-default-prefix.c
new file mode 100644
index 00000000000000..b7a34e2dfa20d4
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/non-default-prefix.c
@@ -0,0 +1,7 @@
+// RUN: %clang_cc1 -verify=check %s
+// RUN: diff %s %s.expected
+void foo() {
+ a = 2; // check-error{{asdf}}
+ // expected-error@-1{ignored}}
+}
+
diff --git a/clang/test/utils/update-verify-tests/Inputs/non-default-prefix.c.expected b/clang/test/utils/update-verify-tests/Inputs/non-default-prefix.c.expected
new file mode 100644
index 00000000000000..087a6be5ea4d2b
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/non-default-prefix.c.expected
@@ -0,0 +1,7 @@
+// RUN: %clang_cc1 -verify=check %s
+// RUN: diff %s %s.expected
+void foo() {
+ a = 2; // check-error{{use of undeclared identifier 'a'}}
+ // expected-error@-1{ignored}}
+}
+
diff --git a/clang/test/utils/update-verify-tests/Inputs/update-same-line.c b/clang/test/utils/update-verify-tests/Inputs/update-same-line.c
new file mode 100644
index 00000000000000..e448800324e373
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/update-same-line.c
@@ -0,0 +1,6 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ bar = 2; // expected-error {{asdf}}
+}
+
diff --git a/clang/test/utils/update-verify-tests/Inputs/update-same-line.c.expected b/clang/test/utils/update-verify-tests/Inputs/update-same-line.c.expected
new file mode 100644
index 00000000000000..ba4387a445fe08
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/update-same-line.c.expected
@@ -0,0 +1,6 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ bar = 2; // expected-error {{use of undeclared identifier 'bar'}}
+}
+
diff --git a/clang/test/utils/update-verify-tests/Inputs/update-single-check.c b/clang/test/utils/update-verify-tests/Inputs/update-single-check.c
new file mode 100644
index 00000000000000..e093a8c2ddddbb
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/update-single-check.c
@@ -0,0 +1,6 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ // expected-error@+1{{asdf}}
+ bar = 2;
+}
diff --git a/clang/test/utils/update-verify-tests/Inputs/update-single-check.c.expected b/clang/test/utils/update-verify-tests/Inputs/update-single-check.c.expected
new file mode 100644
index 00000000000000..42c98d4f77b6ce
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/update-single-check.c.expected
@@ -0,0 +1,6 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ // expected-error@+1{{use of undeclared identifier 'bar'}}
+ bar = 2;
+}
diff --git a/clang/test/utils/update-verify-tests/LitTests/.gitignore b/clang/test/utils/update-verify-tests/LitTests/.gitignore
new file mode 100644
index 00000000000000..307a6d636e7a28
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/LitTests/.gitignore
@@ -0,0 +1,4 @@
+*.c
+*.c.expected
+.lit_test_times.txt
+Output
diff --git a/clang/test/utils/update-verify-tests/LitTests/lit.cfg b/clang/test/utils/update-verify-tests/LitTests/lit.cfg
new file mode 100644
index 00000000000000..dbb22f13ef1303
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/LitTests/lit.cfg
@@ -0,0 +1,31 @@
+import lit.formats
+import lit.llvm
+lit.llvm.initialize(lit_config, config)
+from lit.llvm import llvm_config
+from os.path import dirname
+
+config.name = "Lit Update Verify Tests"
+config.test_format = lit.formats.ShTest(not llvm_config.use_lit_shell)
+config.suffixes = [
+ ".c",
+]
+config.test_source_root = dirname(__file__)
+config.target_triple = None
+config.host_triple = None
+
+config.llvm_tools_dir = lit_config.path[0] # --path is explicitly passed by lit-plugin.test
+llvm_config.use_default_substitutions()
+llvm_config.use_clang()
+config.substitutions.append(("%clang_cc1", "%clang -cc1"))
+
+if lit_config.update_tests:
+ import sys
+ import os
+
+ config.clang_src_dir = dirname(dirname(dirname(dirname(config.test_source_root))))
+ utilspath = os.path.join(config.clang_src_dir, "utils")
+ sys.path.append(utilspath)
+ from UpdateVerifyTests.litplugin import verify_test_updater
+ # normally we'd append to the existing list, but when testing
+ # verify_test_updater we don't want diff_test_updater to accidentally interfere
+ lit_config.test_updaters = [verify_test_updater]
diff --git a/clang/test/utils/update-verify-tests/duplicate-diag.test b/clang/test/utils/update-verify-tests/duplicate-diag.test
new file mode 100644
index 00000000000000..3163ce46199c3f
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/duplicate-diag.test
@@ -0,0 +1,4 @@
+# RUN: cp %S/Inputs/duplicate-diag.c %t.c && not %clang_cc1 -verify %t.c 2>&1 | %update-verify-tests
+# RUN: diff -u %S/Inputs/duplicate-diag.c.expected %t.c
+# RUN: %clang_cc1 -verify %t.c
+
diff --git a/clang/test/utils/update-verify-tests/infer-indentation.test b/clang/test/utils/update-verify-tests/infer-indentation.test
new file mode 100644
index 00000000000000..6ba2f5d9d505bf
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/infer-indentation.test
@@ -0,0 +1,3 @@
+# RUN: cp %S/Inputs/infer-indentation.c %t.c && not %clang_cc1 -verify %t.c 2>&1 | %update-verify-tests
+# RUN: diff -u %S/Inputs/infer-indentation.c.expected %t.c
+# RUN: %clang_cc1 -verify %t.c
diff --git a/clang/test/utils/update-verify-tests/leave-existing-diags.test b/clang/test/utils/update-verify-tests/leave-existing-diags.test
new file mode 100644
index 00000000000000..cde690ef715a67
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/leave-existing-diags.test
@@ -0,0 +1,4 @@
+# RUN: cp %S/Inputs/leave-existing-diags.c %t.c && not %clang_cc1 -verify %t.c 2>&1 | %update-verify-tests
+# RUN: diff -u %S/Inputs/leave-existing-diags.c.expected %t.c
+# RUN: %clang_cc1 -verify %t.c
+
diff --git a/clang/test/utils/update-verify-tests/lit-plugin.test b/clang/test/utils/update-verify-tests/lit-plugin.test
new file mode 100644
index 00000000000000..f3bbf5ec36bfb5
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/lit-plugin.test
@@ -0,0 +1,5 @@
+# RUN: cp %S/Inputs/*.c %S/LitTests
+# RUN: cp %S/Inputs/*.c.expected %S/LitTests
+# RUN: not %{lit} %S/LitTests
+# RUN: not %{lit} %S/LitTests --update-tests
+# RUN: %{lit} %S/LitTests
diff --git a/clang/test/utils/update-verify-tests/lit.local.cfg b/clang/test/utils/update-verify-tests/lit.local.cfg
new file mode 100644
index 00000000000000..281c5be37a983a
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/lit.local.cfg
@@ -0,0 +1,36 @@
+import lit.util
+
+# python 2.7 backwards compatibility
+try:
+ from shlex import quote as shell_quote
+except ImportError:
+ from pipes import quote as shell_quote
+
+if config.standalone_build:
+ # These tests require the update-verify-tests.py script from the clang
+ # source tree, so skip these tests if we are doing standalone builds.
+ config.unsupported = True
+else:
+ config.suffixes = [".test"]
+ config.excludes.add("LitTests")
+
+ script_path = os.path.join(
+ config.clang_src_dir, "utils", "update-verify-tests.py"
+ )
+ python = shell_quote(config.python_executable)
+ config.substitutions.append(
+ (
+ "%update-verify-tests",
+ "%s %s" % (python, shell_quote(script_path)),
+ )
+ )
+ config.substitutions.append(("%{lit}", "%{lit-no-order-opt} --order=lexical"))
+
+ lit_path = os.path.join(config.llvm_tools_dir, "llvm-lit")
+ lit_path = os.path.abspath(lit_path)
+ config.substitutions.append(
+ (
+ "%{lit-no-order-opt}",
+ f"{python} {lit_path} -j1 --path {config.llvm_tools_dir}"
+ )
+)
diff --git a/clang/test/utils/update-verify-tests/multiple-errors.test b/clang/test/utils/update-verify-tests/multiple-errors.test
new file mode 100644
index 00000000000000..1332ef365dc863
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/multiple-errors.test
@@ -0,0 +1,3 @@
+# RUN: cp %S/Inputs/multiple-errors.c %t.c && not %clang_cc1 -verify %t.c 2>&1 | %update-verify-tests
+# RUN: diff -u %S/Inputs/multiple-errors.c.expected %t.c
+# RUN: %clang_cc1 -verify %t.c
diff --git a/clang/test/utils/update-verify-tests/multiple-missing-errors-same-line.test b/clang/test/utils/update-verify-tests/multiple-missing-errors-same-line.test
new file mode 100644
index 00000000000000..a9c21cd77e192b
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/multiple-missing-errors-same-line.test
@@ -0,0 +1,3 @@
+# RUN: cp %S/Inputs/multiple-missing-errors-same-line.c %t.c && not %clang_cc1 -verify %t.c 2>&1 | %update-verify-tests
+# RUN: diff -u %S/Inputs/multiple-missing-errors-same-line.c.expected %t.c
+# RUN: %clang_cc1 -verify %t.c
diff --git a/clang/test/utils/update-verify-tests/no-checks.test b/clang/test/utils/update-verify-tests/no-checks.test
new file mode 100644
index 00000000000000..f6ea91fa552be4
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/no-checks.test
@@ -0,0 +1,3 @@
+# RUN: cp %S/Inputs/no-checks.c %t.c && not %clang_cc1 -verify %t.c 2>&1 | %update-verify-tests
+# RUN: diff -u %S/Inputs/no-checks.c.expected %t.c
+# RUN: %clang_cc1 -verify %t.c
diff --git a/clang/test/utils/update-verify-tests/no-diags.test b/clang/test/utils/update-verify-tests/no-diags.test
new file mode 100644
index 00000000000000..464fe8894253b6
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/no-diags.test
@@ -0,0 +1,4 @@
+# RUN: cp %S/Inputs/no-diag...
[truncated]
|
|
@llvm/pr-subscribers-clang Author: Henrik G. Olsson (hnrklssn) ChangesThis adds a flag to lit for detecting and updating failing tests when possible to do so automatically. The flag uses a plugin architecture where config files can add additional auto-updaters for the types of tests in the test suite. When a test fails with Initially adds support for UpdateVerifyTests and UpdateTestChecks. The flag is currently only implemented for lit's internal shell, so Builds on work in #97369 Patch is 47.68 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/108425.diff 50 Files Affected:
diff --git a/clang/test/Sema/lit.local.cfg b/clang/test/Sema/lit.local.cfg
index baf1b39ef238cd..b385fd92098a8e 100644
--- a/clang/test/Sema/lit.local.cfg
+++ b/clang/test/Sema/lit.local.cfg
@@ -2,3 +2,15 @@ config.substitutions = list(config.substitutions)
config.substitutions.insert(
0, (r"%clang\b", """*** Do not use the driver in Sema tests. ***""")
)
+
+if lit_config.update_tests:
+ import sys
+ import os
+
+ curdir = os.path.dirname(os.path.realpath(__file__))
+ testdir = os.path.dirname(curdir)
+ clangdir = os.path.dirname(testdir)
+ utilspath = os.path.join(clangdir, "utils")
+ sys.path.append(utilspath)
+ from UpdateVerifyTests.litplugin import verify_test_updater
+ lit_config.test_updaters.append(verify_test_updater)
diff --git a/clang/test/lit.cfg.py b/clang/test/lit.cfg.py
index 92a3361ce672e2..c965d3167fe8eb 100644
--- a/clang/test/lit.cfg.py
+++ b/clang/test/lit.cfg.py
@@ -362,3 +362,12 @@ def calculate_arch_features(arch_string):
# possibly be present in system and user configuration files, so disable
# default configs for the test runs.
config.environment["CLANG_NO_DEFAULT_CONFIG"] = "1"
+
+if lit_config.update_tests:
+ import sys
+ import os
+
+ utilspath = os.path.join(config.llvm_src_root, "utils")
+ sys.path.append(utilspath)
+ from update_any_test_checks import utc_lit_plugin
+ lit_config.test_updaters.append(utc_lit_plugin)
diff --git a/clang/test/utils/update-verify-tests/Inputs/duplicate-diag.c b/clang/test/utils/update-verify-tests/Inputs/duplicate-diag.c
new file mode 100644
index 00000000000000..d4a92eb4a7874a
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/duplicate-diag.c
@@ -0,0 +1,10 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ // expected-error@+1{{use of undeclared identifier 'a'}}
+ a = 2; a = 2;
+ b = 2; b = 2;
+ // expected-error@+1 3{{use of undeclared identifier 'c'}}
+ c = 2; c = 2;
+ // expected-error 2{{asdf}}
+}
diff --git a/clang/test/utils/update-verify-tests/Inputs/duplicate-diag.c.expected b/clang/test/utils/update-verify-tests/Inputs/duplicate-diag.c.expected
new file mode 100644
index 00000000000000..5fd72709e487f5
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/duplicate-diag.c.expected
@@ -0,0 +1,10 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ // expected-error@+1 2{{use of undeclared identifier 'a'}}
+ a = 2; a = 2;
+ // expected-error@+1 2{{use of undeclared identifier 'b'}}
+ b = 2; b = 2;
+ // expected-error@+1 2{{use of undeclared identifier 'c'}}
+ c = 2; c = 2;
+}
diff --git a/clang/test/utils/update-verify-tests/Inputs/infer-indentation.c b/clang/test/utils/update-verify-tests/Inputs/infer-indentation.c
new file mode 100644
index 00000000000000..9a58244d92e066
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/infer-indentation.c
@@ -0,0 +1,10 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ // expected-error@+1 2 {{use of undeclared identifier 'a'}}
+ a = 2; a = 2; b = 2; b = 2; c = 2;
+ // expected-error@+1 2 {{asdf}}
+ d = 2;
+ e = 2; f = 2; // expected-error 2 {{use of undeclared identifier 'e'}}
+}
+
diff --git a/clang/test/utils/update-verify-tests/Inputs/infer-indentation.c.expected b/clang/test/utils/update-verify-tests/Inputs/infer-indentation.c.expected
new file mode 100644
index 00000000000000..127efb003b1883
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/infer-indentation.c.expected
@@ -0,0 +1,13 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ // expected-error@+3 {{use of undeclared identifier 'c'}}
+ // expected-error@+2 2 {{use of undeclared identifier 'b'}}
+ // expected-error@+1 2 {{use of undeclared identifier 'a'}}
+ a = 2; a = 2; b = 2; b = 2; c = 2;
+ // expected-error@+1 {{use of undeclared identifier 'd'}}
+ d = 2;
+ // expected-error@+1 {{use of undeclared identifier 'f'}}
+ e = 2; f = 2; // expected-error {{use of undeclared identifier 'e'}}
+}
+
diff --git a/clang/test/utils/update-verify-tests/Inputs/leave-existing-diags.c b/clang/test/utils/update-verify-tests/Inputs/leave-existing-diags.c
new file mode 100644
index 00000000000000..075d6c80fd4c36
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/leave-existing-diags.c
@@ -0,0 +1,13 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ a = 2;
+ // expected-error@-1{{use of undeclared identifier 'a'}}
+ b = 2;// expected-error{{use of undeclared identifier 'b'}}
+ c = 2;
+ // expected-error@7{{use of undeclared identifier 'c'}}
+ d = 2; // expected-error-re{{use of {{.*}} identifier 'd'}}
+
+ e = 2; // error to trigger mismatch
+}
+
diff --git a/clang/test/utils/update-verify-tests/Inputs/leave-existing-diags.c.expected b/clang/test/utils/update-verify-tests/Inputs/leave-existing-diags.c.expected
new file mode 100644
index 00000000000000..778ebd4ff4c7b9
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/leave-existing-diags.c.expected
@@ -0,0 +1,14 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ a = 2;
+ // expected-error@-1{{use of undeclared identifier 'a'}}
+ b = 2;// expected-error{{use of undeclared identifier 'b'}}
+ c = 2;
+ // expected-error@7{{use of undeclared identifier 'c'}}
+ d = 2; // expected-error-re{{use of {{.*}} identifier 'd'}}
+
+ // expected-error@+1{{use of undeclared identifier 'e'}}
+ e = 2; // error to trigger mismatch
+}
+
diff --git a/clang/test/utils/update-verify-tests/Inputs/multiple-errors.c b/clang/test/utils/update-verify-tests/Inputs/multiple-errors.c
new file mode 100644
index 00000000000000..d3fa93ec0e8221
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/multiple-errors.c
@@ -0,0 +1,8 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ a = 2;
+ b = 2;
+
+ c = 2;
+}
diff --git a/clang/test/utils/update-verify-tests/Inputs/multiple-errors.c.expected b/clang/test/utils/update-verify-tests/Inputs/multiple-errors.c.expected
new file mode 100644
index 00000000000000..aa0cf22c8e8a7b
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/multiple-errors.c.expected
@@ -0,0 +1,11 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ // expected-error@+1{{use of undeclared identifier 'a'}}
+ a = 2;
+ // expected-error@+1{{use of undeclared identifier 'b'}}
+ b = 2;
+
+ // expected-error@+1{{use of undeclared identifier 'c'}}
+ c = 2;
+}
diff --git a/clang/test/utils/update-verify-tests/Inputs/multiple-missing-errors-same-line.c b/clang/test/utils/update-verify-tests/Inputs/multiple-missing-errors-same-line.c
new file mode 100644
index 00000000000000..9c5104e693ae57
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/multiple-missing-errors-same-line.c
@@ -0,0 +1,10 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ a = 2; b = 2; c = 2;
+}
+
+void bar() {
+ x = 2; y = 2; z = 2;
+ // expected-error@-1{{use of undeclared identifier 'x'}}
+}
diff --git a/clang/test/utils/update-verify-tests/Inputs/multiple-missing-errors-same-line.c.expected b/clang/test/utils/update-verify-tests/Inputs/multiple-missing-errors-same-line.c.expected
new file mode 100644
index 00000000000000..01076f76318997
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/multiple-missing-errors-same-line.c.expected
@@ -0,0 +1,15 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ // expected-error@+3{{use of undeclared identifier 'c'}}
+ // expected-error@+2{{use of undeclared identifier 'b'}}
+ // expected-error@+1{{use of undeclared identifier 'a'}}
+ a = 2; b = 2; c = 2;
+}
+
+void bar() {
+ x = 2; y = 2; z = 2;
+ // expected-error@-1{{use of undeclared identifier 'x'}}
+ // expected-error@-2{{use of undeclared identifier 'y'}}
+ // expected-error@-3{{use of undeclared identifier 'z'}}
+}
diff --git a/clang/test/utils/update-verify-tests/Inputs/no-checks.c b/clang/test/utils/update-verify-tests/Inputs/no-checks.c
new file mode 100644
index 00000000000000..019698122c3f2b
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/no-checks.c
@@ -0,0 +1,5 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ bar = 2;
+}
diff --git a/clang/test/utils/update-verify-tests/Inputs/no-checks.c.expected b/clang/test/utils/update-verify-tests/Inputs/no-checks.c.expected
new file mode 100644
index 00000000000000..42c98d4f77b6ce
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/no-checks.c.expected
@@ -0,0 +1,6 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ // expected-error@+1{{use of undeclared identifier 'bar'}}
+ bar = 2;
+}
diff --git a/clang/test/utils/update-verify-tests/Inputs/no-diags.c b/clang/test/utils/update-verify-tests/Inputs/no-diags.c
new file mode 100644
index 00000000000000..fddc2ad661fd2d
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/no-diags.c
@@ -0,0 +1,7 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ // expected-error@+1{{asdf}}
+ int a = 2;
+}
+
diff --git a/clang/test/utils/update-verify-tests/Inputs/no-diags.c.expected b/clang/test/utils/update-verify-tests/Inputs/no-diags.c.expected
new file mode 100644
index 00000000000000..acb02badc5655a
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/no-diags.c.expected
@@ -0,0 +1,7 @@
+// expected-no-diagnostics
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ int a = 2;
+}
+
diff --git a/clang/test/utils/update-verify-tests/Inputs/no-expected-diags.c b/clang/test/utils/update-verify-tests/Inputs/no-expected-diags.c
new file mode 100644
index 00000000000000..e02b63dcd7a21b
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/no-expected-diags.c
@@ -0,0 +1,6 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+// expected-no-diagnostics
+void foo() {
+ a = 2;
+}
diff --git a/clang/test/utils/update-verify-tests/Inputs/no-expected-diags.c.expected b/clang/test/utils/update-verify-tests/Inputs/no-expected-diags.c.expected
new file mode 100644
index 00000000000000..7fd85f2e064e28
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/no-expected-diags.c.expected
@@ -0,0 +1,6 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ // expected-error@+1{{use of undeclared identifier 'a'}}
+ a = 2;
+}
diff --git a/clang/test/utils/update-verify-tests/Inputs/non-default-prefix.c b/clang/test/utils/update-verify-tests/Inputs/non-default-prefix.c
new file mode 100644
index 00000000000000..b7a34e2dfa20d4
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/non-default-prefix.c
@@ -0,0 +1,7 @@
+// RUN: %clang_cc1 -verify=check %s
+// RUN: diff %s %s.expected
+void foo() {
+ a = 2; // check-error{{asdf}}
+ // expected-error@-1{ignored}}
+}
+
diff --git a/clang/test/utils/update-verify-tests/Inputs/non-default-prefix.c.expected b/clang/test/utils/update-verify-tests/Inputs/non-default-prefix.c.expected
new file mode 100644
index 00000000000000..087a6be5ea4d2b
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/non-default-prefix.c.expected
@@ -0,0 +1,7 @@
+// RUN: %clang_cc1 -verify=check %s
+// RUN: diff %s %s.expected
+void foo() {
+ a = 2; // check-error{{use of undeclared identifier 'a'}}
+ // expected-error@-1{ignored}}
+}
+
diff --git a/clang/test/utils/update-verify-tests/Inputs/update-same-line.c b/clang/test/utils/update-verify-tests/Inputs/update-same-line.c
new file mode 100644
index 00000000000000..e448800324e373
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/update-same-line.c
@@ -0,0 +1,6 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ bar = 2; // expected-error {{asdf}}
+}
+
diff --git a/clang/test/utils/update-verify-tests/Inputs/update-same-line.c.expected b/clang/test/utils/update-verify-tests/Inputs/update-same-line.c.expected
new file mode 100644
index 00000000000000..ba4387a445fe08
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/update-same-line.c.expected
@@ -0,0 +1,6 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ bar = 2; // expected-error {{use of undeclared identifier 'bar'}}
+}
+
diff --git a/clang/test/utils/update-verify-tests/Inputs/update-single-check.c b/clang/test/utils/update-verify-tests/Inputs/update-single-check.c
new file mode 100644
index 00000000000000..e093a8c2ddddbb
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/update-single-check.c
@@ -0,0 +1,6 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ // expected-error@+1{{asdf}}
+ bar = 2;
+}
diff --git a/clang/test/utils/update-verify-tests/Inputs/update-single-check.c.expected b/clang/test/utils/update-verify-tests/Inputs/update-single-check.c.expected
new file mode 100644
index 00000000000000..42c98d4f77b6ce
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/Inputs/update-single-check.c.expected
@@ -0,0 +1,6 @@
+// RUN: %clang_cc1 -verify %s
+// RUN: diff %s %s.expected
+void foo() {
+ // expected-error@+1{{use of undeclared identifier 'bar'}}
+ bar = 2;
+}
diff --git a/clang/test/utils/update-verify-tests/LitTests/.gitignore b/clang/test/utils/update-verify-tests/LitTests/.gitignore
new file mode 100644
index 00000000000000..307a6d636e7a28
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/LitTests/.gitignore
@@ -0,0 +1,4 @@
+*.c
+*.c.expected
+.lit_test_times.txt
+Output
diff --git a/clang/test/utils/update-verify-tests/LitTests/lit.cfg b/clang/test/utils/update-verify-tests/LitTests/lit.cfg
new file mode 100644
index 00000000000000..dbb22f13ef1303
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/LitTests/lit.cfg
@@ -0,0 +1,31 @@
+import lit.formats
+import lit.llvm
+lit.llvm.initialize(lit_config, config)
+from lit.llvm import llvm_config
+from os.path import dirname
+
+config.name = "Lit Update Verify Tests"
+config.test_format = lit.formats.ShTest(not llvm_config.use_lit_shell)
+config.suffixes = [
+ ".c",
+]
+config.test_source_root = dirname(__file__)
+config.target_triple = None
+config.host_triple = None
+
+config.llvm_tools_dir = lit_config.path[0] # --path is explicitly passed by lit-plugin.test
+llvm_config.use_default_substitutions()
+llvm_config.use_clang()
+config.substitutions.append(("%clang_cc1", "%clang -cc1"))
+
+if lit_config.update_tests:
+ import sys
+ import os
+
+ config.clang_src_dir = dirname(dirname(dirname(dirname(config.test_source_root))))
+ utilspath = os.path.join(config.clang_src_dir, "utils")
+ sys.path.append(utilspath)
+ from UpdateVerifyTests.litplugin import verify_test_updater
+ # normally we'd append to the existing list, but when testing
+ # verify_test_updater we don't want diff_test_updater to accidentally interfere
+ lit_config.test_updaters = [verify_test_updater]
diff --git a/clang/test/utils/update-verify-tests/duplicate-diag.test b/clang/test/utils/update-verify-tests/duplicate-diag.test
new file mode 100644
index 00000000000000..3163ce46199c3f
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/duplicate-diag.test
@@ -0,0 +1,4 @@
+# RUN: cp %S/Inputs/duplicate-diag.c %t.c && not %clang_cc1 -verify %t.c 2>&1 | %update-verify-tests
+# RUN: diff -u %S/Inputs/duplicate-diag.c.expected %t.c
+# RUN: %clang_cc1 -verify %t.c
+
diff --git a/clang/test/utils/update-verify-tests/infer-indentation.test b/clang/test/utils/update-verify-tests/infer-indentation.test
new file mode 100644
index 00000000000000..6ba2f5d9d505bf
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/infer-indentation.test
@@ -0,0 +1,3 @@
+# RUN: cp %S/Inputs/infer-indentation.c %t.c && not %clang_cc1 -verify %t.c 2>&1 | %update-verify-tests
+# RUN: diff -u %S/Inputs/infer-indentation.c.expected %t.c
+# RUN: %clang_cc1 -verify %t.c
diff --git a/clang/test/utils/update-verify-tests/leave-existing-diags.test b/clang/test/utils/update-verify-tests/leave-existing-diags.test
new file mode 100644
index 00000000000000..cde690ef715a67
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/leave-existing-diags.test
@@ -0,0 +1,4 @@
+# RUN: cp %S/Inputs/leave-existing-diags.c %t.c && not %clang_cc1 -verify %t.c 2>&1 | %update-verify-tests
+# RUN: diff -u %S/Inputs/leave-existing-diags.c.expected %t.c
+# RUN: %clang_cc1 -verify %t.c
+
diff --git a/clang/test/utils/update-verify-tests/lit-plugin.test b/clang/test/utils/update-verify-tests/lit-plugin.test
new file mode 100644
index 00000000000000..f3bbf5ec36bfb5
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/lit-plugin.test
@@ -0,0 +1,5 @@
+# RUN: cp %S/Inputs/*.c %S/LitTests
+# RUN: cp %S/Inputs/*.c.expected %S/LitTests
+# RUN: not %{lit} %S/LitTests
+# RUN: not %{lit} %S/LitTests --update-tests
+# RUN: %{lit} %S/LitTests
diff --git a/clang/test/utils/update-verify-tests/lit.local.cfg b/clang/test/utils/update-verify-tests/lit.local.cfg
new file mode 100644
index 00000000000000..281c5be37a983a
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/lit.local.cfg
@@ -0,0 +1,36 @@
+import lit.util
+
+# python 2.7 backwards compatibility
+try:
+ from shlex import quote as shell_quote
+except ImportError:
+ from pipes import quote as shell_quote
+
+if config.standalone_build:
+ # These tests require the update-verify-tests.py script from the clang
+ # source tree, so skip these tests if we are doing standalone builds.
+ config.unsupported = True
+else:
+ config.suffixes = [".test"]
+ config.excludes.add("LitTests")
+
+ script_path = os.path.join(
+ config.clang_src_dir, "utils", "update-verify-tests.py"
+ )
+ python = shell_quote(config.python_executable)
+ config.substitutions.append(
+ (
+ "%update-verify-tests",
+ "%s %s" % (python, shell_quote(script_path)),
+ )
+ )
+ config.substitutions.append(("%{lit}", "%{lit-no-order-opt} --order=lexical"))
+
+ lit_path = os.path.join(config.llvm_tools_dir, "llvm-lit")
+ lit_path = os.path.abspath(lit_path)
+ config.substitutions.append(
+ (
+ "%{lit-no-order-opt}",
+ f"{python} {lit_path} -j1 --path {config.llvm_tools_dir}"
+ )
+)
diff --git a/clang/test/utils/update-verify-tests/multiple-errors.test b/clang/test/utils/update-verify-tests/multiple-errors.test
new file mode 100644
index 00000000000000..1332ef365dc863
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/multiple-errors.test
@@ -0,0 +1,3 @@
+# RUN: cp %S/Inputs/multiple-errors.c %t.c && not %clang_cc1 -verify %t.c 2>&1 | %update-verify-tests
+# RUN: diff -u %S/Inputs/multiple-errors.c.expected %t.c
+# RUN: %clang_cc1 -verify %t.c
diff --git a/clang/test/utils/update-verify-tests/multiple-missing-errors-same-line.test b/clang/test/utils/update-verify-tests/multiple-missing-errors-same-line.test
new file mode 100644
index 00000000000000..a9c21cd77e192b
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/multiple-missing-errors-same-line.test
@@ -0,0 +1,3 @@
+# RUN: cp %S/Inputs/multiple-missing-errors-same-line.c %t.c && not %clang_cc1 -verify %t.c 2>&1 | %update-verify-tests
+# RUN: diff -u %S/Inputs/multiple-missing-errors-same-line.c.expected %t.c
+# RUN: %clang_cc1 -verify %t.c
diff --git a/clang/test/utils/update-verify-tests/no-checks.test b/clang/test/utils/update-verify-tests/no-checks.test
new file mode 100644
index 00000000000000..f6ea91fa552be4
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/no-checks.test
@@ -0,0 +1,3 @@
+# RUN: cp %S/Inputs/no-checks.c %t.c && not %clang_cc1 -verify %t.c 2>&1 | %update-verify-tests
+# RUN: diff -u %S/Inputs/no-checks.c.expected %t.c
+# RUN: %clang_cc1 -verify %t.c
diff --git a/clang/test/utils/update-verify-tests/no-diags.test b/clang/test/utils/update-verify-tests/no-diags.test
new file mode 100644
index 00000000000000..464fe8894253b6
--- /dev/null
+++ b/clang/test/utils/update-verify-tests/no-diags.test
@@ -0,0 +1,4 @@
+# RUN: cp %S/Inputs/no-diag...
[truncated]
|
|
Note that the first 4 commits are from #97369; only the last 2 are new. I'll merge that tomorrow when I have more time to be ready to revert in case something should happen. |
|
✅ With the latest revision this PR passed the Python code formatter. |
hnrklssn
force-pushed
the
lit-update-tests
branch
3 times, most recently
from
b0cd104 to
d41e710
Compare
nhaehnle
reviewed
Sep 14, 2024
nhaehnle
reviewed
Sep 17, 2024
|
I mentioned on a separate commit here, but as a reviewer/code-owner, I very much dislike the existence of tooling like this. I believe it to be actively harmful to the project. |
|
As update-verify-tests was reverted, can you rebase this to just the update_test_checks support? On the LLVM side, we definitely want to have this feature. |
This adds a flag to lit for detecting and updating failing tests when possible to do so automatically. The flag uses a plugin architecture where config files can add additional auto-updaters for the types of tests in the test suite. When a test fails with --update-tests enabled lit passes the test RUN invocation and output to each registered test updater until one of them signals that it updated the test. As such it is the responsibility of the test updater to only update tests where it is reasonably certain that it will actually fix the test, or come close to doing so.
Adds support for invoking the appropriate update_*_test_checks.py script from lit. Checks the header comment for which script was used to generate it in the first place, so only test cases that were already generated are affected. To support this the interface for test updater functions is expanded to not only take a ShellCommandResult, but also the Test object. This makes it easy to get the file path of the current test. Also adds a --path flag to update_any_test_checks.py as a convenience to avoid having to manually set the PATH variable.
hnrklssn
force-pushed
the
lit-update-tests
branch
from
db9bd0a to
20d7d62
Compare
Done
and done |
nikic
approved these changes
Oct 1, 2024
hnrklssn
added a commit
that referenced
this pull request
Oct 2, 2024
puja2196
pushed a commit
to puja2196/LLVM-tutorial
that referenced
this pull request
Oct 2, 2024
Sterling-Augustine
pushed a commit
to Sterling-Augustine/llvm-project
that referenced
this pull request
Oct 3, 2024
This adds a flag to lit for detecting and updating failing tests when possible to do so automatically. The flag uses a plugin architecture where config files can add additional auto-updaters for the types of tests in the test suite. When a test fails with `--update-tests` enabled lit passes the test RUN invocation and output to each registered test updater until one of them signals that it updated the test (or all test updaters have been run). As such it is the responsibility of the test updater to only update tests where it is reasonably certain that it will actually fix the test, or come close to doing so. Initially adds support for UpdateVerifyTests and UpdateTestChecks. The flag is currently only implemented for lit's internal shell, so `--update-tests` implies `LIT_USE_INTERNAL_SHELL=1`. Builds on work in llvm#97369 Fixes llvm#81320
Sterling-Augustine
pushed a commit
to Sterling-Augustine/llvm-project
that referenced
this pull request
Oct 3, 2024
schwitanski
pushed a commit
to RWTH-HPC/llvm-lit-mirror
that referenced
this pull request
Feb 27, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds a flag to lit for detecting and updating failing tests when possible to do so automatically. The flag uses a plugin architecture where config files can add additional auto-updaters for the types of tests in the test suite. When a test fails with
--update-testsenabled lit passes the test RUN invocation and output to each registered test updater until one of them signals that it updated the test (or all test updaters have been run). As such it is the responsibility of the test updater to only update tests where it is reasonably certain that it will actually fix the test, or come close to doing so.Initially adds support for UpdateVerifyTests and UpdateTestChecks. The flag is currently only implemented for lit's internal shell, so
--update-testsimpliesLIT_USE_INTERNAL_SHELL=1.Builds on work in #97369
Fixes #81320