support PGO on custom project

[csmoe]

make PGO easier for custom toolchain distribution.

r? @Kobzol

[rustbot]

Failed to set assignee to Kobzol: invalid assignee

Note: Only org members, users with write permissions, or people who have commented on the PR may be assigned.

[Kobzol]

Hi, the use-case seems reasonable. Although care should be taken that the custom command will generate the profiles into the correct files - you will thus probably want to distinguish between gathering profiles for LLVM/Rustc/BOLT even for custom runners, and pass the correct env. variables to them.

Therefore, I would prefer a more "OOP-like" implementation, to reduce complexity and conditions in the logic of the individual stages, and also to move the various benchmark logic to a single place.

Something like

class BenchmarkRunner:
    def run_rustc(self, pipeline: Pipeline):
    def run_llvm(self, pipeline: Pipeline):
    def run_bolt(self, pipeline: Pipeline):

    # (or maybe just a single method that would receive an enum - Rustc/LLVM/BOLT)

class DefaultBenchmarkRunner(BenchmarkRunner):
    def run_rustc(self, pipeline: Pipeline):
         self.run_compiler_benchmarks(
            pipeline,
            profiles=["Check", "Debug", "Opt"],
            scenarios=["All"],
            crates=RUSTC_PGO_CRATES,
            env=dict(
                LLVM_PROFILE_FILE=str(pipeline.rustc_profile_template_path())
            )
        )
   ...


class CustomBenchmarkRunner(BenchmarkRunner):
    ...

and then some function that creates the corresponding BenchmarkRunner from the environment at the beginning of the script. The selected runner would be then passed to the individual stages (it could also be stored within Pipeline directly to make it easier to access, it's not super nice because of the cyclic dependency, but it should be fine here).

[Kobzol]

Good, this is how I imagined it :) Is this enough for you? Or do you also want to implement support for custom runners in this PR?

[csmoe]

Really thanks for your kind review.
I haven't written much python, I'm imaging a common workflow as:

# content of a custom build.py

from stage-build import BenchmarkRunner, run
class CustomRunner(BenchmarkRunner):
    ...
runner = CustomRunner()
run(runner)

So I move the logic in __main__ into a run function for external invocation, is that ok?

[Kobzol]

Alright, that sounds reasonable :) Let's see if everything still works.

@bors try @rust-timer queue

[bors]

⌛ Trying commit 27beb46 with merge 47ee1f814e2773f0403b768ccb711b607af2b80d...

[bors]

☀️ Try build successful - checks-actions
Build commit: 47ee1f814e2773f0403b768ccb711b607af2b80d (47ee1f814e2773f0403b768ccb711b607af2b80d)

[rust-timer]

Finished benchmarking commit (47ee1f814e2773f0403b768ccb711b607af2b80d): comparison URL.

Overall result: no relevant changes - no action needed

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	3.8%	[3.8%, 3.8%]	1
Regressions ❌ (secondary)	3.1%	[3.1%, 3.1%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	3.8%	[3.8%, 3.8%]	1

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 660.274s -> 659.215s (-0.16%)

[Kobzol]

Perf looks good, LGTM.

I don't have merge right, so reassigning.

r? @Mark-Simulacrum

[Mark-Simulacrum]

@bors r=Kobzol

This seems fine to me, though of course no stability guarantees are offered on these scripts.

[bors]

📌 Commit 27beb46 has been approved by Kobzol

It is now in the queue for this repository.

[bors]

⌛ Testing commit 27beb46 with merge ba6f5e3...

[bors]

☀️ Test successful - checks-actions
Approved by: Kobzol
Pushing ba6f5e3 to master...

[rust-timer]

Finished benchmarking commit (ba6f5e3): comparison URL.

Overall result: ✅ improvements - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.3%	[-0.3%, -0.3%]	1
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.3%	[2.0%, 2.6%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-1.3%	[-1.7%, -1.0%]	4
All ❌✅ (primary)	-	-	0

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	1.9%	[1.3%, 3.4%]	5
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 641.167s -> 643.087s (0.30%)