Skip to content

Make untracked incr. comp. information inaccessible #90317

@cjgillot

Description

@cjgillot
Contributor

When iterating over a collection, be it a Vec, a HashMap or an IndexMap, the order of items influences the value of the resulting hash: [a, b] and [b, a] have different hashes.

Meanwhile, there is some information we do not want to track. This is the case of the value of LocalDefId. Inserting a function in a file will change other functions LocalDefId, but it will not change their DefPathHash.

My concern is about controlling this information flow. In order to do that ToStableHashKey trait replaces the iteration order of the collection (which for hash maps is based on the key and the allocated memory capacity and should be irrelevant to the compilation), by the order of a stable hash key (the DefPathHash when the key is LocalDefId). By sorting the vectors by stable key, we manage the information flow.

Using IndexMap, the iteration order is the insertion order. Normally, this insertion order should only depend on tracked information obtained by depending on another query. For instance, a HIR visitor will create a query dependency on hir_owner_nodes, which hashes the in-code declaration order of functions. However, and this is my concern, the order of LocalDefId is freely available without using a query and is purposely untracked.

In order to make IndexMaps safe for incr. comp., we need to ensure untracked information is inaccessible.

Known issues:

  • Stop implementing PartialOrd and Ord on DefId;
    Stop implementing PartialOrd, Ord and Idx for LocalDefId;
    Stop implementing PartialOrd, Ord and Idx for LocalExpnId;
    Stop implementing PartialOrd and Ord for SyntaxContext;
    Enforce that UNTRACKED options are not accessed within a query.

Originally posted by @cjgillot in #90253 (comment)

Activity

added
A-incr-compArea: Incremental compilation
E-hardCall for participation: Hard difficulty. Experience needed to fix: A lot.
E-help-wantedCall for participation: Help is requested to fix this issue.
on Oct 26, 2021
pierwill

pierwill commented on Oct 29, 2021

@pierwill
Member
  • Stop implementing PartialOrd and Ord on DefId...

Can these items be done one at a time?

cjgillot

cjgillot commented on Oct 29, 2021

@cjgillot
ContributorAuthor

Of course!

pierwill

pierwill commented on Oct 29, 2021

@pierwill
Member

Will give this a go and see what happens. @rustbot claim

pierwill

pierwill commented on Oct 30, 2021

@pierwill
Member

I started experimenting with LocalDefId. I wonder if the DefId work might end up being slightly less complex. 🤔

pierwill

pierwill commented on Nov 2, 2021

@pierwill
Member

I wonder if the DefId work might end up being slightly less complex. 🤔

Update: No. 😄

pierwill

pierwill commented on Nov 3, 2021

@pierwill
Member

Looks like removing Ord, PartialOrd for ExpnId and LocalExpnId might require changes to the macro rustc_index::newtype_index because of this line:

#[derive(Copy, PartialEq, Eq, Hash, PartialOrd, Ord, $($derives),*)]

bjorn3

bjorn3 commented on Nov 5, 2021

@bjorn3
Member

Track accesses to command-line options;
Track accesses to debugging command-line options.

Most of them are already "tracked" by discarding the entire incr comp cache if they change. The ones that aren't tracked are explicitly not tracked. See the UNTRACKED mentions in https://github.com/rust-lang/rust/blob/d22dd65835190278f315e06442614142653ec98f/compiler/rustc_session/src/options.rs A couple of them should be marked as TRACKED though I think.

pierwill

pierwill commented on Nov 8, 2021

@pierwill

115 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

A-incr-compArea: Incremental compilationE-hardCall for participation: Hard difficulty. Experience needed to fix: A lot.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @oli-obk@Aaron1011@cjgillot@bjorn3@pierwill

      Issue actions

        Make untracked incr. comp. information inaccessible · Issue #90317 · rust-lang/rust