New `ot-tidy.py` script for development and CI #192

ziuziakowska · 2025-09-23T09:48:02Z

This replaces the existing ot-tidy.sh script with a new Python script.

This new script should make it easier to run the linter locally in the same way as CI, as the older wrapper script did not have any usage information.

The script also checks for some misconfigurations and reports those - such as QEMU not being configured to use Clang which results in a different incompatible set of warning flags to be used with GCC to be used for clang-tidy.

In addition, this script adds a -j flag to run linting in parallel, using the run-clang-tidy wrapper that comes with clang-tidy. On my machine, this reduced linting time on the same set of files as CI from ~3 minutes to 30 seconds.

jwnrt

I get this error from the script:

$ scripts/opentitan/ot_tidy.py --ci-files
Could not parse 'clang-tidy' version string.

Here's my output of clang-tidy --version:

$ clang-tidy --version
LLVM (http://llvm.org/):
  LLVM version 16.0.2
  Optimized build.

I think it's because my version_line is the second line, not the first

ziuziakowska · 2025-09-23T10:10:50Z

I think it's because my version_line is the second line, not the first

Fixed it to look for that on any line.
Mine appeared like this:

$ clang-tidy-20 --version
Ubuntu LLVM version 20.1.2
  Optimized build.

jwnrt · 2025-09-23T10:16:26Z

Yep, that works now, thanks

rivos-eblot

I think several features are broken, and some are a bit fragile (parsing, ANSI management, ...).

Can you run flake8, as there are many outsized lines and argument aligment warning messages. Thanks.

scripts/opentitan/ot_tidy.py

.github/workflows/build_test.yaml

rivos-eblot · 2025-09-23T14:14:20Z

scripts/opentitan/ot_tidy.py

+    parser.add_argument('-f', '--files', required=False, nargs="+", action='extend', default=[],
+        help="Files to run tidy on")
+
+    args = parser.parse_args()


It think there is a regression here (vs. .sh script): we need to be able to pass any argument to clang-tidy, not just files.

Is this used anywhere? I'm not sure how compatible it would be, since run-clang-tidy and clang-tidy support different flags, and the former doesn't have a way to pass arguments to the latter. Both only have the option to pass additional flags to the compiler with -extra-arg/-extra-arg-before and -- [ARGS...] respectively.

Yes we do use it: the public version of OpenTitan emulation is only about a half of what we emulate: we wrote ot-tidy.sh so that it could be use by either public OpenTitan or with our extensions. Our CI extensively uses it.

I wonder whether it would make sense to have two tidy helper scripts, as this one is getting complex and is likely to become more difficult to use at least from a developer from the command line. Maybe a dedicated script to run run-clang-tidy would be easier, and keep ot-tidy.sh as it is? ot-tidy.sh is really though as a very thin wrapper around cland-tidy: it clang-tidy with some sugar syntax.

I see. Perhaps it might make more sense to have separate helper scripts then. I originally had the thought to write this because the only usage information was in the CI, and the warning flag errors when not configuring with --cc=clang were only documented in the comments there too. The run-clang-tidy wrapper also let me lint a lot faster instead of waiting for CI to do it. So I suppose this script is more public development-oriented.

out of curiosity: does run-clang-tidy manage output streams from its various subprocesses when several of them report errors?

scripts/opentitan/ot_tidy.py

docs/opentitan/tools.md

scripts/opentitan/ot_tidy.py

This script is intendde to replace the `ot-tidy.sh` script for running clang-tidy on our part of the codebase. A `-j` flag is added to run clang-tidy jobs in parallel, using the run-clang-tidy script that comes with it. On my machine this speeds up linting files for CI from >3 minutes to ~30 seconds. The script also checks for some configuration issues, such as the C compiler for QEMU not being set to clang - this causes some warnings flags to be enabled that are recognised by GCC but not clang, and these will show up as errors. Signed-off-by: Alice Ziuziakowska <[email protected]>

Replaces all usages, updates docs, and removes the older script. Signed-off-by: Alice Ziuziakowska <[email protected]>

rivos-eblot · 2025-09-24T14:18:05Z

scripts/opentitan/ot-tidy.py

+    try:
+        parser = argparse.ArgumentParser()
+
+        parser.add_argument("-b", "--build-dir", required=False,


You can remove the required=False which is the default behavior for all options

rivos-eblot · 2025-09-24T14:26:06Z

scripts/opentitan/ot_tidy.py

+    parser.add_argument('-f', '--files', required=False, nargs="+", action='extend', default=[],
+        help="Files to run tidy on")
+
+    args = parser.parse_args()


out of curiosity: does run-clang-tidy manage output streams from its various subprocesses when several of them report errors?

rivos-eblot · 2025-09-25T09:54:44Z

scripts/opentitan/ot-tidy.py

+                            action="store_true",
+                            help="Run on all files checked by CI")
+
+        parser.add_argument("-f", "--files", required=False,


I wonder if this case is allowed: required=False means the option is optional, but on the other side nargs="+" indicates that the option should be specified at least once.

BTW, maybe this which is the main one should use positional argument (files) rather than an option?

rivos-eblot · 2025-09-25T11:20:41Z

scripts/opentitan/ot-tidy.py

+        if args.ci_files:
+            args.files.extend(ci_files())
+
+        if not args.files:


The option you specify to the parser should handle this case I believe

rivos-eblot · 2025-09-25T11:21:16Z

scripts/opentitan/ot-tidy.py

+            ]
+
+        fmt_args = " ".join(cmd_args[1:])
+        print(f"run '{cmd} {fmt_args} ... ({len(args.files)} files)'")


Maybe use a logger for this?

rivos-eblot · 2025-09-25T11:22:06Z

scripts/opentitan/ot-tidy.py

+    eprint = eprint_tty
+
+
+def check_compile_commands(build_dir) -> None:


missing type hint (: str)

rivos-eblot · 2025-09-25T11:23:36Z

scripts/opentitan/ot-tidy.py

+
+eprint = eprint_plain
+if sys.stderr.isatty():
+    eprint = eprint_tty


or

eprint = eprint_tty if sys.stderr.isatty() else eprint_plain

rivos-eblot · 2025-09-25T11:24:10Z

scripts/opentitan/ot-tidy.py

+    return [relpath(normpath(join(QEMU_ROOT, f))) for f in ci_files_list]
+
+
+def uint(value) -> int:


rivos-eblot · 2025-09-25T11:25:08Z

scripts/opentitan/ot-tidy.py

+LLVM_MAJOR = 20
+"""LLVM major version to check for"""
+
+OT_SCRIPTS = dirname(normpath(__file__))


my mistake from a previous comment: abspath is better than normpath, sorry about that.

rivos-eblot · 2025-09-25T11:28:49Z

scripts/opentitan/ot-tidy.py

+            if match is not None:
+                ccs = [f"clang-{LLVM_MAJOR}", "clang"]
+                if match.group(1) not in ccs:
+                    eprint("QEMU build is configured with non-clang C compiler "


May be use f"" on each line, it should be more readable.
Python would only complain if the resulting string contains no variable.

jwnrt self-requested a review September 23, 2025 09:52

ziuziakowska force-pushed the ot-tidy branch from 6f186e8 to 6a440b3 Compare September 23, 2025 09:56

jwnrt reviewed Sep 23, 2025

View reviewed changes

ziuziakowska force-pushed the ot-tidy branch from 6a440b3 to 898b946 Compare September 23, 2025 10:09

ziuziakowska force-pushed the ot-tidy branch from 898b946 to 8cb2bfe Compare September 23, 2025 10:20

ziuziakowska marked this pull request as ready for review September 23, 2025 11:25

ziuziakowska requested review from AlexJones0 and rivos-eblot September 23, 2025 11:45

rivos-eblot requested changes Sep 23, 2025

View reviewed changes

rivos-eblot reviewed Sep 23, 2025

View reviewed changes

scripts/opentitan/ot_tidy.py Outdated Show resolved Hide resolved

rivos-eblot reviewed Sep 23, 2025

View reviewed changes

scripts/opentitan/ot_tidy.py Outdated Show resolved Hide resolved

rivos-eblot reviewed Sep 23, 2025

View reviewed changes

scripts/opentitan/ot_tidy.py Outdated Show resolved Hide resolved

ziuziakowska force-pushed the ot-tidy branch from 8cb2bfe to 700b982 Compare September 24, 2025 13:44

ziuziakowska requested review from jwnrt and rivos-eblot September 24, 2025 13:46

[ot] ci: .github,.gitlab-ci.d: replace ot-tidy.sh with ot-tidy.py

665fef0

Replaces all usages, updates docs, and removes the older script. Signed-off-by: Alice Ziuziakowska <[email protected]>

ziuziakowska force-pushed the ot-tidy branch from 700b982 to 665fef0 Compare September 24, 2025 13:57

ziuziakowska changed the title ~~New ot_tidy.py script for development and CI~~ New ot-tidy.py script for development and CI Sep 24, 2025

rivos-eblot requested changes Sep 25, 2025

View reviewed changes

		eprint = eprint_tty


		def check_compile_commands(build_dir) -> None:

		return [relpath(normpath(join(QEMU_ROOT, f))) for f in ci_files_list]


		def uint(value) -> int:

New ot-tidy.py script for development and CI #192

Are you sure you want to change the base?

New ot-tidy.py script for development and CI #192

Uh oh!

Conversation

ziuziakowska commented Sep 23, 2025

Uh oh!

jwnrt left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ziuziakowska commented Sep 23, 2025

Uh oh!

jwnrt commented Sep 23, 2025

Uh oh!

rivos-eblot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rivos-eblot Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ziuziakowska Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

New `ot-tidy.py` script for development and CI #192

New `ot-tidy.py` script for development and CI #192

jwnrt left a comment •

edited

Loading

rivos-eblot left a comment •

edited

Loading

rivos-eblot Sep 24, 2025 •

edited

Loading

ziuziakowska Sep 24, 2025 •

edited

Loading