From cd057f4d94221a145d1f12fa2505f99a72b73583 Mon Sep 17 00:00:00 2001 From: pierwill Date: Thu, 17 Feb 2022 13:54:20 -0600 Subject: [PATCH 01/11] Split sentence --- src/query.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/query.md b/src/query.md index 719568791..840c29f07 100644 --- a/src/query.md +++ b/src/query.md @@ -11,7 +11,7 @@ some item, will compute the type of that item and return it to you. [def-id]: appendix/glossary.md#def-id [hl]: ./compiler-src.md -Query execution is **memoized** – so the first time you invoke a +Query execution is **memoized**. The first time you invoke a query, it will go do the computation, but the next time, the result is returned from a hashtable. Moreover, query execution fits nicely into **incremental computation**; the idea is roughly that, when you do a From 1e89dee95dda533d9be3b65c382d0be2a57002e0 Mon Sep 17 00:00:00 2001 From: pierwill Date: Thu, 17 Feb 2022 13:56:24 -0600 Subject: [PATCH 02/11] Move up instructions for invoking a query --- src/query.md | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/src/query.md b/src/query.md index 840c29f07..14901b091 100644 --- a/src/query.md +++ b/src/query.md @@ -35,24 +35,26 @@ will in turn demand information about that crate, starting from the However, that vision is not fully realized. Still, big chunks of the compiler (for example, generating MIR) work exactly like this. -### Incremental Compilation in Detail - -The [Incremental Compilation in Detail][query-model] chapter gives a more -in-depth description of what queries are and how they work. -If you intend to write a query of your own, this is a good read. - -[query-model]: queries/incremental-compilation-in-detail.md - ### Invoking queries -To invoke a query is simple. The tcx ("type context") offers a method -for each defined query. So, for example, to invoke the `type_of` +Invoking a query is simple. The [`TyTcx`] ("type context") struct offers a method +for each defined query. For example, to invoke the `type_of` query, you would just do this: ```rust,ignore let ty = tcx.type_of(some_def_id); ``` +[`TyTcx`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/ty/struct.TyCtxt.html + +### Incremental Compilation in Detail + +The [Incremental Compilation in Detail][query-model] chapter gives a more +in-depth description of what queries are and how they work. +If you intend to write a query of your own, this is a good read. + +[query-model]: queries/incremental-compilation-in-detail.md + ### How the compiler executes a query So you may be wondering what happens when you invoke a query From 0b1f54e90f8bf9394087e623a5c7cf62b645bdb3 Mon Sep 17 00:00:00 2001 From: pierwill Date: Thu, 17 Feb 2022 14:10:18 -0600 Subject: [PATCH 03/11] More significant edits --- src/query.md | 44 +++++++++++++++++++++----------------------- 1 file changed, 21 insertions(+), 23 deletions(-) diff --git a/src/query.md b/src/query.md index 14901b091..87c494d8e 100644 --- a/src/query.md +++ b/src/query.md @@ -2,29 +2,31 @@ As described in [the high-level overview of the compiler][hl], the Rust compiler is still (as of July 2021) transitioning from a -traditional "pass-based" setup to a "demand-driven" system. **The Compiler Query -System is the key to our new demand-driven organization.** The idea is pretty -simple. You have various queries that compute things about the input – for -example, there is a query called `type_of(def_id)` that, given the [def-id] of +traditional "pass-based" setup to a "demand-driven" system. The compiler query +system is the key to rustc's demand-driven organization. +The idea is pretty simple. Instead of entirely independent passes +(parsing, type-checking, etc.), various smaller *queries* +compute information about the input source. For example, +there is a query called `type_of(def_id)` that, given the [`DefId`] of some item, will compute the type of that item and return it to you. -[def-id]: appendix/glossary.md#def-id +[`DefId`]: appendix/glossary.md#def-id [hl]: ./compiler-src.md -Query execution is **memoized**. The first time you invoke a +Query execution is *memoized*. The first time you invoke a query, it will go do the computation, but the next time, the result is returned from a hashtable. Moreover, query execution fits nicely into -**incremental computation**; the idea is roughly that, when you do a -query, the result **may** be returned to you by loading stored data -from disk (but that's a separate topic we won't discuss further here). +*incremental computation*; the idea is roughly that, when you do a +query, the result *may* be returned to you by loading stored data +from disk.[^1] -The overall vision is that, eventually, the entire compiler -control-flow will be query driven. There will effectively be one -top-level query ("compile") that will run compilation on a crate; this +Eventually, we want the entire compiler +control-flow to be query driven. There will effectively be one +top-level query (`compile`) that will run compilation on a crate; this will in turn demand information about that crate, starting from the *end*. For example: -- This "compile" query might demand to get a list of codegen-units +- The `compile` query might demand to get a list of codegen-units (i.e. modules that need to be compiled by LLVM). - But computing the list of codegen-units would invoke some subquery that returns the list of all modules defined in the Rust source. @@ -32,8 +34,8 @@ will in turn demand information about that crate, starting from the - This keeps going further and further back until we wind up doing the actual parsing. -However, that vision is not fully realized. Still, big chunks of the -compiler (for example, generating MIR) work exactly like this. +Although this vision is not fully realized, large sections of +compiler (for example, generating [MIR](mir/)) currently work exactly like this. ### Invoking queries @@ -47,14 +49,6 @@ let ty = tcx.type_of(some_def_id); [`TyTcx`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/ty/struct.TyCtxt.html -### Incremental Compilation in Detail - -The [Incremental Compilation in Detail][query-model] chapter gives a more -in-depth description of what queries are and how they work. -If you intend to write a query of your own, this is a good read. - -[query-model]: queries/incremental-compilation-in-detail.md - ### How the compiler executes a query So you may be wondering what happens when you invoke a query @@ -310,3 +304,7 @@ More discussion and issues: [GitHub issue #42633]: https://github.com/rust-lang/rust/issues/42633 [Incremental Compilation Beta]: https://internals.rust-lang.org/t/incremental-compilation-beta/4721 [Incremental Compilation Announcement]: https://blog.rust-lang.org/2016/09/08/incremental.html + +[^1]: The ["Incremental Compilation in Detail](queries/incremental-compilation-in-detail.md) chapter gives a more +in-depth description of what queries are and how they work. +If you intend to write a query of your own, this is a good read. From 816c2298d8b98912dddf45c2b256eb52ecd105dd Mon Sep 17 00:00:00 2001 From: pierwill Date: Thu, 17 Feb 2022 14:19:50 -0600 Subject: [PATCH 04/11] Start editing "Adding a new query" section --- src/query.md | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/src/query.md b/src/query.md index 87c494d8e..ed4e02e65 100644 --- a/src/query.md +++ b/src/query.md @@ -1,5 +1,7 @@ # Queries: demand-driven compilation + + As described in [the high-level overview of the compiler][hl], the Rust compiler is still (as of July 2021) transitioning from a traditional "pass-based" setup to a "demand-driven" system. The compiler query @@ -158,13 +160,13 @@ they define both a `provide` and a `provide_extern` function, through [rustc_metadata]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_metadata/index.html [wasm_import_module_map]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_codegen_ssa/back/symbol_export/fn.wasm_import_module_map.html -### Adding a new kind of query +### Adding a new query -So suppose you want to add a new kind of query, how do you do so? -Well, defining a query takes place in two steps: +Suppose you want to add a new query: how do you do so? +Defining a query takes place in two steps: -1. first, you have to specify the query name and arguments; and then, -2. you have to supply query providers where needed. +1. Specify the query name and its arguments. +2. Supply query providers where needed. To specify the query name and arguments, you simply add an entry to the big macro invocation in @@ -186,8 +188,9 @@ rustc_queries! { ``` Queries are grouped into categories (`Other`, `Codegen`, `TypeChecking`, etc.). -Each group contains one or more queries. Each query definition is broken up like -this: +Each group contains one or more queries. + +A query definition has the following form: ```rust,ignore query type_of(key: DefId) -> Ty<'tcx> { ... } @@ -200,7 +203,7 @@ query type_of(key: DefId) -> Ty<'tcx> { ... } query keyword ``` -Let's go over them one by one: +Let's go over these elements one by one: - **Query keyword:** indicates a start of a query definition. - **Name of query:** the name of the query method @@ -213,11 +216,7 @@ Let's go over them one by one: - **Result type of query:** the type produced by this query. This type should (a) not use `RefCell` or other interior mutability and (b) be cheaply cloneable. Interning or using `Rc` or `Arc` is recommended for - non-trivial data types. - - The one exception to those rules is the `ty::steal::Steal` type, - which is used to cheaply modify MIR in place. See the definition - of `Steal` for more details. New uses of `Steal` should **not** be - added without alerting `@rust-lang/compiler`. + non-trivial data types.[^2] - **Query modifiers:** various flags and options that customize how the query is processed (mostly with respect to [incremental compilation][incrcomp]). @@ -308,3 +307,8 @@ More discussion and issues: [^1]: The ["Incremental Compilation in Detail](queries/incremental-compilation-in-detail.md) chapter gives a more in-depth description of what queries are and how they work. If you intend to write a query of your own, this is a good read. + +[^2]: The one exception to those rules is the `ty::steal::Steal` type, +which is used to cheaply modify MIR in place. See the definition +of `Steal` for more details. New uses of `Steal` should **not** be +added without alerting `@rust-lang/compiler`. From 8523378d0fdf0115db5fee37c699faeeeba4814f Mon Sep 17 00:00:00 2001 From: pierwill Date: Thu, 17 Feb 2022 14:23:09 -0600 Subject: [PATCH 05/11] Edit query structs section --- src/query.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/query.md b/src/query.md index ed4e02e65..43d45aaeb 100644 --- a/src/query.md +++ b/src/query.md @@ -231,9 +231,9 @@ So, to add a query: #### Query structs and descriptions -For each kind, the `rustc_queries` macro will generate a "query struct" +For each query, the `rustc_queries` macro will generate a "query struct" named after the query. This struct is a kind of a place-holder -describing the query. Each such struct implements the +describing the query. Each query struct implements the [`self::config::QueryConfig`][QueryConfig] trait, which has associated types for the key/value of that particular query. Basically the code generated looks something like this: From be6ffbf2bcc15ab62f985ca1a4f884a7b70f4cb1 Mon Sep 17 00:00:00 2001 From: pierwill <19642016+pierwill@users.noreply.github.com> Date: Thu, 17 Feb 2022 15:00:52 -0600 Subject: [PATCH 06/11] Apply suggestions from code review Co-authored-by: Noah Lev --- src/query.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/query.md b/src/query.md index 43d45aaeb..96f144ce2 100644 --- a/src/query.md +++ b/src/query.md @@ -18,7 +18,7 @@ some item, will compute the type of that item and return it to you. Query execution is *memoized*. The first time you invoke a query, it will go do the computation, but the next time, the result is returned from a hashtable. Moreover, query execution fits nicely into -*incremental computation*; the idea is roughly that, when you do a +*incremental computation*; the idea is roughly that, when you invoke a query, the result *may* be returned to you by loading stored data from disk.[^1] @@ -36,12 +36,12 @@ will in turn demand information about that crate, starting from the - This keeps going further and further back until we wind up doing the actual parsing. -Although this vision is not fully realized, large sections of -compiler (for example, generating [MIR](mir/)) currently work exactly like this. +Although this vision is not fully realized, large sections of the +compiler (for example, generating [MIR](./mir/)) currently work exactly like this. ### Invoking queries -Invoking a query is simple. The [`TyTcx`] ("type context") struct offers a method +Invoking a query is simple. The [`TyCtxt`] ("type context") struct offers a method for each defined query. For example, to invoke the `type_of` query, you would just do this: @@ -162,7 +162,7 @@ they define both a `provide` and a `provide_extern` function, through ### Adding a new query -Suppose you want to add a new query: how do you do so? +How do you add a new query? Defining a query takes place in two steps: 1. Specify the query name and its arguments. @@ -232,7 +232,7 @@ So, to add a query: #### Query structs and descriptions For each query, the `rustc_queries` macro will generate a "query struct" -named after the query. This struct is a kind of a place-holder +named after the query. This struct is a kind of placeholder describing the query. Each query struct implements the [`self::config::QueryConfig`][QueryConfig] trait, which has associated types for the key/value of that particular query. Basically the code generated looks something From 171f7ac1e525b2cae7a8ca6cd2302bfba6081be3 Mon Sep 17 00:00:00 2001 From: pierwill Date: Thu, 17 Feb 2022 15:03:32 -0600 Subject: [PATCH 07/11] Rename and move footnotes --- src/query.md | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/src/query.md b/src/query.md index 96f144ce2..6fc31d3fb 100644 --- a/src/query.md +++ b/src/query.md @@ -20,7 +20,7 @@ query, it will go do the computation, but the next time, the result is returned from a hashtable. Moreover, query execution fits nicely into *incremental computation*; the idea is roughly that, when you invoke a query, the result *may* be returned to you by loading stored data -from disk.[^1] +from disk.[^incr-comp-detail] Eventually, we want the entire compiler control-flow to be query driven. There will effectively be one @@ -39,6 +39,10 @@ will in turn demand information about that crate, starting from the Although this vision is not fully realized, large sections of the compiler (for example, generating [MIR](./mir/)) currently work exactly like this. +[^incr-comp-detail]: The ["Incremental Compilation in Detail](queries/incremental-compilation-in-detail.md) chapter gives a more +in-depth description of what queries are and how they work. +If you intend to write a query of your own, this is a good read. + ### Invoking queries Invoking a query is simple. The [`TyCtxt`] ("type context") struct offers a method @@ -216,7 +220,7 @@ Let's go over these elements one by one: - **Result type of query:** the type produced by this query. This type should (a) not use `RefCell` or other interior mutability and (b) be cheaply cloneable. Interning or using `Rc` or `Arc` is recommended for - non-trivial data types.[^2] + non-trivial data types.[^steal] - **Query modifiers:** various flags and options that customize how the query is processed (mostly with respect to [incremental compilation][incrcomp]). @@ -229,6 +233,11 @@ So, to add a query: - Link the provider by modifying the appropriate `provide` method; or add a new one if needed and ensure that `rustc_driver` is invoking it. +[^steal]: The one exception to those rules is the `ty::steal::Steal` type, +which is used to cheaply modify MIR in place. See the definition +of `Steal` for more details. New uses of `Steal` should **not** be +added without alerting `@rust-lang/compiler`. + #### Query structs and descriptions For each query, the `rustc_queries` macro will generate a "query struct" @@ -304,11 +313,3 @@ More discussion and issues: [Incremental Compilation Beta]: https://internals.rust-lang.org/t/incremental-compilation-beta/4721 [Incremental Compilation Announcement]: https://blog.rust-lang.org/2016/09/08/incremental.html -[^1]: The ["Incremental Compilation in Detail](queries/incremental-compilation-in-detail.md) chapter gives a more -in-depth description of what queries are and how they work. -If you intend to write a query of your own, this is a good read. - -[^2]: The one exception to those rules is the `ty::steal::Steal` type, -which is used to cheaply modify MIR in place. See the definition -of `Steal` for more details. New uses of `Steal` should **not** be -added without alerting `@rust-lang/compiler`. From 99fb398a664394b0a55cbb4af7942cb9e2fef1e2 Mon Sep 17 00:00:00 2001 From: pierwill <19642016+pierwill@users.noreply.github.com> Date: Thu, 17 Feb 2022 15:05:52 -0600 Subject: [PATCH 08/11] Update src/query.md Co-authored-by: Noah Lev --- src/query.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/query.md b/src/query.md index 6fc31d3fb..191c41a94 100644 --- a/src/query.md +++ b/src/query.md @@ -7,7 +7,7 @@ is still (as of July 2021) transitioning from a traditional "pass-based" setup to a "demand-driven" system. The compiler query system is the key to rustc's demand-driven organization. The idea is pretty simple. Instead of entirely independent passes -(parsing, type-checking, etc.), various smaller *queries* +(parsing, type-checking, etc.), a set of function-like *queries* compute information about the input source. For example, there is a query called `type_of(def_id)` that, given the [`DefId`] of some item, will compute the type of that item and return it to you. From 46a290a3aeba767e4813b3919fcfbb627b76da8a Mon Sep 17 00:00:00 2001 From: pierwill <19642016+pierwill@users.noreply.github.com> Date: Thu, 17 Feb 2022 15:06:02 -0600 Subject: [PATCH 09/11] Update src/query.md Co-authored-by: Noah Lev --- src/query.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/query.md b/src/query.md index 191c41a94..f5e28704f 100644 --- a/src/query.md +++ b/src/query.md @@ -9,7 +9,7 @@ system is the key to rustc's demand-driven organization. The idea is pretty simple. Instead of entirely independent passes (parsing, type-checking, etc.), a set of function-like *queries* compute information about the input source. For example, -there is a query called `type_of(def_id)` that, given the [`DefId`] of +there is a query called `type_of` that, given the [`DefId`] of some item, will compute the type of that item and return it to you. [`DefId`]: appendix/glossary.md#def-id From 57d78f8d372488ef3a36ce34e990564e33c283eb Mon Sep 17 00:00:00 2001 From: pierwill Date: Thu, 17 Feb 2022 15:08:42 -0600 Subject: [PATCH 10/11] Link to docs --- src/query.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/query.md b/src/query.md index f5e28704f..780394d6b 100644 --- a/src/query.md +++ b/src/query.md @@ -12,7 +12,7 @@ compute information about the input source. For example, there is a query called `type_of` that, given the [`DefId`] of some item, will compute the type of that item and return it to you. -[`DefId`]: appendix/glossary.md#def-id +[`DefId`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/def_id/struct.DefId.html [hl]: ./compiler-src.md Query execution is *memoized*. The first time you invoke a From eba8b64f09e5a5ff50f6c7e858f17b337e637f3e Mon Sep 17 00:00:00 2001 From: pierwill Date: Thu, 17 Feb 2022 15:09:51 -0600 Subject: [PATCH 11/11] Edit query diagram --- src/query.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/query.md b/src/query.md index 780394d6b..9c8cbd9b9 100644 --- a/src/query.md +++ b/src/query.md @@ -198,10 +198,10 @@ A query definition has the following form: ```rust,ignore query type_of(key: DefId) -> Ty<'tcx> { ... } -^^ ^^^^^^^ ^^^^^ ^^^^^^^^ ^^^ +^^^^^ ^^^^^^^ ^^^^^ ^^^^^^^^ ^^^ | | | | | | | | | query modifiers -| | | result type of query +| | | result type | | query key type | name of query query keyword