Merge branch 'main' into ber.a/diagnosticData

Author: vzarytovskii

docs/changing-the-ast.md

@@ -0,0 +1,87 @@
+---
+title: Changing the AST
+category: Compiler Internals
+categoryindex: 200
+index: 800
+---
+# Changing the AST
+
+Making changes to the AST is a common task when working on new F# compiler features or when working on developer tooling.  
+This document describes the process of making changes to the AST.
+
+The easiest way to modify the AST is to start with the type definitions in `SyntaxTree.fsi` and `SyntaxTree.fs` and then let the compiler guide you to the places where you need to make changes.
+Let's look at an example: We want to extend the AST to include the range of the `/` symbol in a `SynRationalConst.Rational`.  
+
+There are two solutions to choose from:
+- Add a new field to the `Rational` union case
+- Add a dedicated trivia type to the union case which contains the new range and maybe move the existing ranges to the trivia type as well  
+
+The pros of introducing a dedicated trivia type are:
+- Having the additional information in a separate structure allows it to grow more easily over time. Adding new information to an existing trivia type won't disrupt most FCS consumers.  
+- It is clear that it is information that is not relevant for the compilation.  
+
+The cons are: 
+- It can be a bit excessive to introduce for a single field.
+- The existing AST node might already contain fields that are historically more suited for trivia, but predate the SyntaxTrivia module.
+
+In this example, we'll go with the first solution and add a new field named `divRange` to the `Rational` union case as it felt a bit excessive to introduce a new trivia type for a single field.  
+But these are the type of decisions that need to be made when changing the AST.
+
+```fsharp
+type SynRationalConst =
+
+    // ...
+
+    | Rational of
+        numerator: int32 *
+        numeratorRange: range *
+        divRange: range *   // our new field
+        denominator: int32 *
+        denominatorRange: range *
+        range: range
+    
+    // ...
+```	
+
+After modifying `SyntaxTree.fsi` and `SyntaxTree.fs`, the compiler will report erros in `pars.fsy`. If not, the `fsy` file wasn't processed by the compilation. In this case, a rebuild of `FSharp.Compiler.Service.fsproj` should help.  
+`pars.fsy` is the parser specification of F#, a list of rules that describe how to parse F# code. Don't be scared by the size of the file or the unfamiliar content.
+It's easier than it looks.
+The F# compiler uses a parser generator called [fsyacc](https://github.com/fsprojects/FsLexYacc) to generate the parser from the specification in `pars.fsy`.
+Let's look at the most relevant syntax parts of a `.fsy` file:
+
+```fsharp
+rationalConstant:
+  | INT32 INFIX_STAR_DIV_MOD_OP INT32
+    { if $2 <> "/" then reportParseErrorAt (rhs parseState 2) (FSComp.SR.parsUnexpectedOperatorForUnitOfMeasure())
+      if fst $3 = 0 then reportParseErrorAt (rhs parseState 3) (FSComp.SR.parsIllegalDenominatorForMeasureExponent())
+      if (snd $1) || (snd $3) then errorR(Error(FSComp.SR.lexOutsideThirtyTwoBitSigned(), lhs parseState))
+      SynRationalConst.Rational(fst $1, rhs parseState 1, fst $3, rhs parseState 3, lhs parseState) }
+  | // ...
+```
+
+The first line is the name of the rule, `rationalConstant` in this case. It's a so called non-terminal symbol in contrast to a terminal symbol like `INT32` or `INFIX_STAR_DIV_MOD_OP`. The individual cases of the rule are separated by `|`, they are called productions.
+
+By now, you should be able to see the similarities between an fsyacc rule and the pattern matching you know from F#.  
+The code between the curly braces is the code that gets executed when the rule is matched and is _real_ F# code. After compilation, it ends up in 
+`.\artifacts\obj\FSharp.Compiler.Service\Debug\netstandard2.0\pars.fs`, generated by fsyacc.
+
+The first three lines do error checking and report errors if the input is invalid.
+Then the code calls the `Rational` constructor of `SynRationalConst` and passes some values to it. Here we need to make changes to adjust the parser to our modified type definition.  
+The values or symbols that matched the rule are available as `$1`, `$2`, `$3` etc. in the code. As you can see, `$1` is a tuple, consisting of the parsed number and a boolean indicating whether the number is a valid 32 bit signed integer or not.
+The code is executed in the context of the parser, so you can use the `parseState` variable, an instance of `IParseState`, to access the current state of the parser. There are helper functions defined in `ParseHelpers.fs` that make it easier to work with it.  
+`rhs parseState 1` returns the range of the first symbol that matched the rule, here `INT32`. So, it returns the range of `23` in `23/42`.  
+`rhs` is short for _right hand side_.  
+Another helper is `rhs2`. Using it like `rhs2 parseState 2 3` for example, returns the range covering the symbols from the second to the third symbol that matched the rule. Given `23/42`, it would return the range of `/42`.  
+ `lhs parseState` returns the range of the whole rule, `23/42` in our example.
+ When parser recovery is of concern for a rule, it's preferred to use `rhs2` over `lhs`.
+ 
+ Circling back to our original example of adding a new field to `SynRationalConst`, we need to add a new parameter to the call of the `Rational` constructor. We want to pass the range of the `/` symbol, so we need to add `rhs parseState 2` as the third parameter to the constructor call:  
+ 
+ ```fsharp
+SynRationalConst.Rational(fst $1, rhs parseState 1, rhs parseState 2, fst $3, rhs parseState 3, lhs parseState)
+```	
+
+That's it. Adjusting the other constructor calls of `Rational` in `pars.fsy` should be enough to have a working parser again which returns the modified AST.  
+While fixing the remaining compiler errors outside of `pars.fsy`, it's a good idea to use named access to the fields of the `SynRationalConst.Rational` union case instead of positional access. This way, the compilation won't fail if aditional fields are added to the union case in the future.  
+After a successful compilation, you can run the parser tests in `SyntaxTreeTests.fs` to verify that everything works as expected.  
+It's likely that you'll need to update the baseline files as decribed in `SyntaxTreeTests.fs`.

docs/fcs/editor.fsx

@@ -184,7 +184,7 @@ let decls =
 
 // Print the names of available items
 for item in decls.Items do
-    printfn " - %s" item.Name
+    printfn " - %s" item.NameInList
 
 (**
 

docs/fcs/filesystem.fsx

@@ -130,6 +130,9 @@ let B = File1.A + File1.A"""
         member _.IsStableFileHeuristic(path) =
             defaultFileSystem.IsStableFileHeuristic(path)
 
+        member this.ChangeExtensionShim(path, extension) =
+            defaultFileSystem.ChangeExtensionShim(path, extension)
+
 let myFileSystem = MyFileSystem()
 FileSystem <- MyFileSystem()
 

docs/fcs/syntax-visitor.fsx

@@ -0,0 +1,189 @@
+(**
+---
+title: Tutorial: SyntaxVisitorBase
+category: FSharp.Compiler.Service
+categoryindex: 300
+index: 301
+---
+*)
+(*** hide ***)
+#I "../../artifacts/bin/FSharp.Compiler.Service/Debug/netstandard2.0"
+(**
+Compiler Services: Using the SyntaxVisitorBase
+=========================================
+
+Syntax tree traversal is a common topic when interacting with the `FSharp.Compiler.Service`.  
+As established in [Tutorial: Expressions](./untypedtree.html#Walking-over-the-AST), the [ParsedInput](../reference/fsharp-compiler-syntax-parsedinput.html) can be traversed by a set of recursive functions.  
+It can be tedious to always construct these functions from scratch.
+
+As an alternative, a [SyntaxVisitorBase](../reference/fsharp-compiler-syntax-syntaxvisitorbase-1.html) can be used to traverse the syntax tree.
+Consider, the following code sample:
+*)
+
+let codeSample = """
+module Lib
+
+let myFunction paramOne paramTwo =
+    ()
+"""
+
+(**
+Imagine we wish to grab the `myFunction` name from the `headPat` in the [SynBinding](../reference/fsharp-compiler-syntax-synbinding.html).  
+Let's introduce a helper function to construct the AST:
+*)
+
+#r "FSharp.Compiler.Service.dll"
+open FSharp.Compiler.CodeAnalysis
+open FSharp.Compiler.Text
+open FSharp.Compiler.Syntax
+
+let checker = FSharpChecker.Create()
+
+/// Helper to construct an ParsedInput from a code snippet.
+let mkTree codeSample =
+    let parseFileResults =
+        checker.ParseFile(
+            "FileName.fs",
+            SourceText.ofString codeSample,
+            { FSharpParsingOptions.Default with SourceFiles = [| "FileName.fs" |] }
+        )
+        |> Async.RunSynchronously
+
+    parseFileResults.ParseTree
+
+(**
+And create a visitor to traverse the tree:
+*)
+
+let visitor =
+    { new SyntaxVisitorBase<string>() with
+        override this.VisitPat(path, defaultTraverse, synPat) =
+            // First check if the pattern is what we are looking for.
+            match synPat with
+            | SynPat.LongIdent(longDotId = SynLongIdent(id = [ ident ])) ->
+                // Next we can check if the current path of visited nodes, matches our expectations.
+                // The path will contain all the ancestors of the current node.
+                match path with
+                // The parent node of `synPat` should be a `SynBinding`.
+                | SyntaxNode.SynBinding _ :: _ ->
+                    // We return a `Some` option to indicate we found what we are looking for.
+                    Some ident.idText
+                // If the parent is something else, we can skip it here.
+                | _ -> None
+            | _ -> None }
+
+let result = SyntaxTraversal.Traverse(Position.pos0, mkTree codeSample, visitor) // Some "myFunction"
+
+(**
+Instead of traversing manually from `ParsedInput` to `SynModuleOrNamespace` to `SynModuleDecl.Let` to `SynBinding` to `SynPat`, we leverage the default navigation that happens in `SyntaxTraversal.Traverse`.  
+A `SyntaxVisitorBase` will shortcut all other code paths once a single `VisitXYZ` override has found anything.
+
+Our code sample of course only had one let binding and thus we didn't need to specify any further logic whether to differentiate between multiple bindings.
+Let's consider a second example where we know the user's cursor inside an IDE is placed after `c` and we are interested in the body expression of the let binding.
+*)
+
+let secondCodeSample = """
+module X
+
+let a = 0
+let b = 1
+let c = 2
+"""
+
+let secondVisitor =
+    { new SyntaxVisitorBase<SynExpr>() with
+        override this.VisitBinding(path, defaultTraverse, binding) =
+            match binding with
+            | SynBinding(expr = e) -> Some e }
+
+let cursorPos = Position.mkPos 6 5
+
+let secondResult =
+    SyntaxTraversal.Traverse(cursorPos, mkTree secondCodeSample, secondVisitor) // Some (Const (Int32 2, (6,8--6,9)))
+
+(**
+Due to our passed cursor position, we did not need to write any code to exclude the expressions of the other let bindings.
+`SyntaxTraversal.Traverse` will check whether the current position is inside any syntax node before drilling deeper.
+
+Lastly, some `VisitXYZ` overrides can contain a defaultTraverse. This helper allows you to continue the default traversal when you currently hit a node that is not of interest.
+Consider `1 + 2 + 3 + 4`, this will be reflected in a nested infix application expression.
+If the cursor is at the end of the entire expression, we can grab the value of `4` using the following visitor:
+*)
+
+let thirdCodeSample = "let sum = 1 + 2 + 3 + 4"
+
+(*
+AST will look like:
+
+Let
+ (false,
+  [SynBinding
+     (None, Normal, false, false, [],
+      PreXmlDoc ((1,0), Fantomas.FCS.Xml.XmlDocCollector),
+      SynValData
+        (None, SynValInfo ([], SynArgInfo ([], false, None)), None,
+         None),
+      Named (SynIdent (sum, None), false, None, (1,4--1,7)), None,
+      App
+        (NonAtomic, false,
+         App
+           (NonAtomic, true,
+            LongIdent
+              (false,
+               SynLongIdent
+                 ([op_Addition], [], [Some (OriginalNotation "+")]),
+               None, (1,20--1,21)),
+            App
+              (NonAtomic, false,
+               App
+                 (NonAtomic, true,
+                  LongIdent
+                    (false,
+                     SynLongIdent
+                       ([op_Addition], [],
+                        [Some (OriginalNotation "+")]), None,
+                     (1,16--1,17)),
+                  App
+                    (NonAtomic, false,
+                     App
+                       (NonAtomic, true,
+                        LongIdent
+                          (false,
+                           SynLongIdent
+                             ([op_Addition], [],
+                              [Some (OriginalNotation "+")]), None,
+                           (1,12--1,13)),
+                        Const (Int32 1, (1,10--1,11)), (1,10--1,13)),
+                     Const (Int32 2, (1,14--1,15)), (1,10--1,15)),
+                  (1,10--1,17)), Const (Int32 3, (1,18--1,19)),
+               (1,10--1,19)), (1,10--1,21)),
+         Const (Int32 4, (1,22--1,23)), (1,10--1,23)), (1,4--1,7),
+      Yes (1,0--1,23), { LeadingKeyword = Let (1,0--1,3)
+                         InlineKeyword = None
+                         EqualsRange = Some (1,8--1,9) })
+*)
+
+let thirdCursorPos = Position.mkPos 1 22
+
+let thirdVisitor =
+    { new SyntaxVisitorBase<int>() with
+        override this.VisitExpr(path, traverseSynExpr, defaultTraverse, synExpr) =
+            match synExpr with
+            | SynExpr.Const (constant = SynConst.Int32 v) -> Some v
+            // We do want to continue to traverse when nodes like `SynExpr.App` are found.
+            | otherExpr -> defaultTraverse otherExpr }
+
+let thirdResult =
+    SyntaxTraversal.Traverse(cursorPos, mkTree thirdCodeSample, thirdVisitor) // Some 4
+
+(**
+`defaultTraverse` is especially useful when you do not know upfront what syntax tree you will be walking.  
+This is a common case when dealing with IDE tooling. You won't know what actual code the end-user is currently processing.
+
+**Note: SyntaxVisitorBase is designed to find a single value inside a tree!**  
+This is not an ideal solution when you are interested in all nodes of certain shape.  
+It will always verify if the given cursor position is still matching the range of the node.    
+As a fallback the first branch will be explored when you pass `Position.pos0`.  
+By design, it is meant to find a single result.
+
+*)

docs/fcs/tokenizer.fsx

@@ -36,7 +36,7 @@ file name of the source code. The defined symbols are required because the
 tokenizer handles `#if` directives. The file name is required only to specify
 locations of the source code (and it does not have to exist):
 *)
-let sourceTok = FSharpSourceTokenizer([], Some "C:\\test.fsx")
+let sourceTok = FSharpSourceTokenizer([], Some "C:\\test.fsx", Some "PREVIEW", None)
 (**
 Using the `sourceTok` object, we can now (repeatedly) tokenize lines of 
 F# source code.

docs/fcs/untypedtree.fsx

@@ -111,11 +111,11 @@ used more often):
 /// Walk over a pattern - this is for example used in 
 /// let <pat> = <expr> or in the 'match' expression
 let rec visitPattern = function
-  | SynPat.Wild(_) -> 
+  | SynPat.Wild _ -> 
       printfn "  .. underscore pattern"
-  | SynPat.Named(name, _, _, _) ->
+  | SynPat.Named(ident = SynIdent(ident = name)) ->
       printfn "  .. named as '%s'" name.idText
-  | SynPat.LongIdent(LongIdentWithDots(ident, _), _, _, _, _, _, _) ->
+  | SynPat.LongIdent(longDotId = SynLongIdent(id = ident)) ->
       let names = String.concat "." [ for i in ident -> i.idText ]
       printfn "  .. identifier: %s" names
   | pat -> printfn "  .. other pattern: %A" pat
@@ -145,9 +145,7 @@ let rec visitExpression e =
       // for 'let .. = .. and .. = .. in ...'
       printfn "LetOrUse with the following bindings:"
       for binding in bindings do
-        let (SynBinding(
-          access, kind, isInline, isMutable, attrs, xmlDoc, data,
-          headPat, retInfo, init, equalsRange, debugPoint, trivia)) = binding
+        let (SynBinding(headPat = headPat; expr = init)) = binding
         visitPattern headPat
         visitExpression init
       // Visit the body expression
@@ -177,9 +175,7 @@ let visitDeclarations decls =
         // Let binding as a declaration is similar to let binding
         // as an expression (in visitExpression), but has no body
         for binding in bindings do
-          let (SynBinding(
-            access, kind, isInline, isMutable, attrs, xmlDoc, 
-            valData, pat, retInfo, body, equalsRange, debugPoint, trivia)) = binding
+          let (SynBinding(headPat = pat; expr = body)) = binding
           visitPattern pat 
           visitExpression body         
     | _ -> printfn " - not supported declaration: %A" declaration
@@ -194,7 +190,7 @@ with multiple `namespace Foo` declarations:
 /// does not explicitly define it..
 let visitModulesAndNamespaces modulesOrNss =
   for moduleOrNs in modulesOrNss do
-    let (SynModuleOrNamespace(lid, isRec, isMod, decls, xml, attrs, accessibility, range)) = moduleOrNs
+    let (SynModuleOrNamespace(longId = lid; decls = decls)) = moduleOrNs
     printfn "Namespace or module: %A" lid
     visitDeclarations decls
 (**
@@ -238,7 +234,7 @@ in the previous step:
 match tree with
 | ParsedInput.ImplFile(implFile) ->
     // Extract declarations and walk over them
-    let (ParsedImplFileInput(fn, script, name, _, _, modules, _, _)) = implFile
+    let (ParsedImplFileInput(contents = modules)) = implFile
     visitModulesAndNamespaces modules
 | _ -> failwith "F# Interface file (*.fsi) not supported."
 (**

eng/Version.Details.xml

@@ -1,9 +1,9 @@
 <?xml version="1.0" encoding="utf-8"?>
 <Dependencies>
   <ProductDependencies>
-    <Dependency Name="Microsoft.SourceBuild.Intermediate.source-build-reference-packages" Version="8.0.0-alpha.1.23451.1">
+    <Dependency Name="Microsoft.SourceBuild.Intermediate.source-build-reference-packages" Version="8.0.0-alpha.1.23455.3">
       <Uri>https://github.com/dotnet/source-build-reference-packages</Uri>
-      <Sha>0030d238c7929b0e9b06576837b60ad90037b1d2</Sha>
+      <Sha>75ec14a961f43446d952c64b5b3330df750db54f</Sha>
       <SourceBuild RepoName="source-build-reference-packages" ManagedOnly="true" />
     </Dependency>
     <Dependency Name="Microsoft.SourceBuild.Intermediate.msbuild" Version="17.7.0-preview-23217-02">