Implicit Referencing - opting in and opting out

[johnnyreilly]

Hi,

I've a question about Visual Studio's Implicit Referencing. I've looked around in vain for a definitive answer and so I thought it might make sense to raise the question here.

Previously, all TypeScript files in a project had to reference each other explicitly. With 0.9.1, they now implicitly reference all other TypeScript files in the project.

Since 0.9.1 I've used this feature pretty happily and purged my files of the mass of /// <reference path= entries.

I've recently been wondering if there are any hidden "gotchas" to using /// <reference path= in a Visual Studio context? My understanding is that if there are any /// <reference path= instances in a file then implicit referencing goes out the window and only the specified references are used. Is that correct? (So use of any references in the head of a ts file is effectively the same as opting that file out of Implicit Referencing for Visual Studio.)

@mhegazy - this might be a question for you? Similar ground was covered here.

[mhegazy]

Short answer is no. /// references is honored regardless of how the context is built.

/// references are considered "dependencies" and the compiler/Language service will honor these dependencies while building the compilation context. where a compilation context is all files part of a single compilation unit (i.e. a single invocation to tsc.exe). Building a context is similar in all supported modes of interaction with VS and the compiler. Here is how a context is built (bold signals differences).

1. Command line compiler

• Start with files passed on the command line, in order
• For each file, examine its /// references and import = require() statements
  • If the file has not been seen before add it to the list, and recursively examine its references

2. Visual Studio, with a project referencing the file

• Start with all files in the project with build action === "TypeScriptCompile", in order of reference in the project file
• For each file, examine its /// references and import = require() statements
  • If the file has not been seen before add it to the list, and recursively examine its references

3. Visual Studio, with No project referencing the file (loose files)

• Assume all loose files currently open as part of one project "Miscellaneous"
• Start with all open files, that do not belong to a project (see above)
• For each file, examine its /// references and import = require() statements
  • If the file has not been seen before add it to the list, and recursively examine its references

There is not "implicit references" per se, its detecting the "compilation context". In a project the msbuild task will take all the TypeScriptCompile items and pass them to the compiler on the command line, the behavior here should be identical between VS and the command line. With loose files, we have no way of knowing how you are going to build this file, so we make an assumption that all the files currently open are part of one compilation unit and will all be passed on the command line to the compiler.

in all cases, you can use /// references to pull in files into your context that you have not specified earlier.

Now, some caveats:

1. MSBuild Incremental build
   Problem: MSBuild incremental build support relies on checking the time stamp of the inputs against the outputs, if you are in a project context, inputs are all TypeScriptCompiler items, if you have a file that is not references in the project, but pulled in as a /// reference, MSBuild has no way to know about it, so it will not check its time stamp. so if you update only that file, and build, the build will be skipped as MSBuild thinks the project is up-to-date.
   Solution: Include all your files in your project, use /// references for ordering or documenting dependency, but not to pull in new items.
2. Ordering output with --out
   Problem: The compiler will generate outputs in the order of occurrence in the dependency graph. The dependency graph is built using the logic stated earlier. In the batch compiler case, you can change your order on the command line, but in VS, manually editing the project file to enforce order is neither convenient nor productive.
   Solution: There is support to a special file "_references.ts" at the root of your project (has to be at the root, if anywhere else it will be treated like a normal file). If this file exists, the file is guaranteed to be passed as the first file in the list, both in VS and in the MSBuild task. that allows you to specify order through adding /// references in _references.ts. Remember that the graph is built in order, and following /// references in order, so the order of /// references in _references.ts will dictate the order of files in the output, in absence of any circular references.

Diagnosis:

Here are some tips to diagnose problems related to context building:

1. Enable "Virtual Project" view:
   Go to Tools > Options > Text Editor > TypeScript > Project, and enable the two check boxes for Virtual Projects (see screen snapshot)
   
2. Understanding Virtual Projects:
   In the screen shot below, I have 3 files open:
3. app.ts part of a project (TypeScriptHTMLApp14)
4. NotnProject1.ts, pulled in through a /// reference from app.ts
5. NotInProject2.ts, not part of any project, nor referenced from any file that belongs to a project, just open (loose file)
6. NotInProject3.ts, another loose file, but closed.

From the screenshot you can see a new TypeScript Virtual Projects node under the solution. this shows you how VS have computed the contexts. as you can see there are two "Compilation Context" or Projects:

• Project#1 -- corresponds to my real project, TypeScriptHTMLApp14, and was built by first including app.ts, then walking its references to include NotInProject1.ts
• Project#2 -- Miscellaneous, corresponds to all other "loose files" that were not part of a project. We start by open files, NotInProject2.ts and follow its /// references to load a closed file "NotInProject3.ts"

[johnnyreilly]

Hi @mhegazy,

Thank you for the fully detailed response! Very helpful.

First important take home:

• Context is added to not replaced when using /// references. So it's not possible to wipe existing references in the project just by adding a /// reference to the top of a file. Perfectly clear - thanks.

Second important take home:

There is not "implicit references" per se, its detecting the "compilation context"

Thanks for detailing the 3 different ways context can be constructed. That's really helpful. I wasn't aware of the details of the third listed way:

Visual Studio, with No project referencing the file (loose files)

I’d love you to clarify this a little if you could. It sounds like this scenario covers where someone is just using Visual Studio as an IDE for TypeScript. Presumably this means 1 of 2 things; either

1. There is no csproj at all.
2. Or there is a csproj but it does not refererence *.ts files.

Question about 1.

If 1. is the case how does Visual Studio decide the compiler switches for TypeScript? What I mean is, even the Visual Studio TypeScript template solution ships with a csproj file that says <TypeScriptToolsVersion>1.0</TypeScriptToolsVersion>, <TypeScriptRemoveComments>false</TypeScriptRemoveComments> etc. Without out these specified how does tsc get invoked? Just as is? (So specifying –noImplicitAny or similar is not possible in this scenario?)

Either way, the implication is that /// references should be used when using loose *.ts files to ensure context is built correctly.

Question about _references.ts

I know _references.ts was originally introduced to allow the ordering of the files (particularly important for having the correct overload resolution when there's a million jQuery plugins). Presumably in the “no project referencing the file” scenario it could also serve as a replacement for having the csproj listing all the build action === "TypeScriptCompile" files?

What I mean is, you could use _references.ts to control your context construction / implicit referencing. This would be achieved by listing all TS files in _references.ts and either by having every TS file reference _references.ts in turn (or by always keeping _references.ts open in VS – which probably isn’t as practical).

Is there any reason this shouldn't work? I'm assuming that if _references.ts was located at the highest level in the world of scripts it will considered as being in the root and so will still be passed as the first file to tsc.

Question about ASP.Net vNext world

Given that in the ASP.Net vNext world I understand that csproj files are to disappear what is the plan for context building in this sort of project? Is it loose files all the way?

[mhegazy]

Question 1:
That is a great question. In loose files we have no way of knowing how the user is building the files. as you mentioned they could be using it as a text editor, and they have a different way to manage their build. so we make no assumptions. as for configuration, you get the default configurations from tsc.exe with some exceptions:

• target === ES5 // basically the latest
• toolsVersion === Latest // again, latest

There are two options that you can control through tools/options (see screenshot below, namely:

• compile on save
• module (amd or commonjs -- default unspecified)

Note that these configurations only affect loose files. for files in projects, you need to set them for every project.

Question 2:
one note, _references.ts is only special in the project scenario. in the loose file scenario it is just a normal file.
One common practice that we have seen is using an index.ts file that lists all the files in your project, and all the files in your project reference index.ts; this way you open a file and you get the whole project loaded correctly all the time.
you could definitely use _references.ts to do that. my recommendation is not to mix projects and /// references if you can, to ensure your incremental compilation works as expected.

Question 3:
I have not played with the new ASP.net projects in a while. but i do not think there is much change in the fundamentals since i looked at it. The ASP.net project still provide a project system interface to VS, though you do not have a project file. The TypeScript language service asks the project system about the files in the project. as long as the project system gives us the typescript files, we should be good.
The problem is the configuration. but if you are using a build tool like grunt or bower, there is a lot of specific knowlege that needs to be backed into the language service, which i do not think we will be able to support.

[johnnyreilly]

Thanks @mhegazy. That clarifies things a great deal.

my recommendation is not to mix projects and /// references if you can, to ensure your incremental compilation works as expected.

This totally makes sense. However it raises an issue about having a TypeScript project which some people may be using Visual Studio to work on and some people may be using another IDE (eg WebStorm). In this situation it would be necessary to use /// references for the people not using Visual Studio.

In order to avoid confusion in this situation the ideal would seem to be having a way of opting out of VS project support so everyone has the same experience regardless of IDE. So everyone would depend on using /// references to build context.

I don't think there is a way to do this at present. Is this something that could made an option? eg. Have a Visual Studio setting that disabled implicit referencing?

[johnnyreilly]

@jonathandturner did discuss the possibility of providing this option back on Oct 23, 2013 by the way:

https://typescript.codeplex.com/workitem/1471#CommentContainer7

This discussion took a number of different turns after that and the option was never really returned to. Just thought I'd provide this as a reference.

[mhegazy]

@johnnyreilly i am not sure i understand the desired behavior. is the problem the project file? what if you just forget about the project file and open the files in VS and let /// reference resolution figure out your context?

if this is the case, just add after your import to Microsoft.TypeScript.targets. just note that it will ignore your project file all together. that means files, and configuration, and will treat all files as if they did not belong to a project.

  <PropertyGroup>
    <!-- Indicates to the language service that this project supports TypeScript -->
    <TypeScriptEnabled>false</TypeScriptEnabled>
  </PropertyGroup>

[johnnyreilly]

@mhegazy,

I'm guess the question is, what's the best way to set up a TypeScript project that can be worked on regardless of IDE (so whether it's Visual Studio, WebStorm etc)?

As you say, you could have both the project file figuring out context for Visual Studio users and /// references in place for non-Visual Studio users. But I'm mindful that you'll have the situation of trying to keep both the /// references and the TypeScriptCompile references in the project file in line. Also, it's not very DRY 😄 Your comment seems to express a similar concern?:

my recommendation is not to mix projects and /// references if you can, to ensure your incremental compilation works as expected.

For that reason I thought it might be simpler to abandon the project file approach entirely. That way the only maintenance is just the /// references and all developers (regardless of IDE) should need to do the same maintenance to keep the code compiling. No danger that a Visual Studio user might add a file to the project but forget to add the appropriate /// reference if you see what I mean.

I appreciate this might sound like an odd thing to make a fuss about but I'm very keen to reduce "developer friction". When I've been introducing people to TypeScript I've had them complain that they feel the tooling is fighting against them on occasion. I want to reduce that as much as possible. Whilst using the project file to build context is a really handy shortcut for Visual Studio users it also kind of erects a barrier for TypeScript users between the Visual Studio world and those outside it. (I'm sure it wasn't intended to be that - but that's kind of a side effect I guess.)

I'll try out the <TypeScriptEnabled> approach you suggest and report back.

[johnnyreilly]

Hi @mhegazy,

I put together a little test project using the TypeScriptEnabled approach suggested. You can see it here:

https://github.com/johnnyreilly/proverb-without-implicit-referencing

It has the csproj change you suggest:

  <PropertyGroup>
    <!-- Indicates to the language service that this project supports TypeScript -->
    <TypeScriptEnabled>false</TypeScriptEnabled>
  </PropertyGroup>

I created a file I called _references.d.ts which is just a list of the TypeScript files and definition files my project uses:

https://github.com/johnnyreilly/proverb-without-implicit-referencing/blob/master/Proverb.Web/app/_references.d.ts

And each TypeScript file has a /// <reference path="_references.d.ts" /> (with appropriate path tweaks) at it's head. Seems to work just fine. Is this what you were proposing?

So am I right in thinking that <TypeScriptEnabled>false</TypeScriptEnabled> in actual fact means "deactivate implicit referencing" or is there more to it than that? I hadn't heard of this flag before....

[mhegazy]

@johnnyreilly, increasing developer productivity is the reason typescript exists, so friction is not something desired.

The part that I find confusing is why you need the project file. If you just open one of your files by itself, /// references will cause you to load the all your files, similar to other editors. I am assuming you are using gulp to build anyways.

Either ways, figuring out files is not the only issue, compiler options are the other piece of the puzzle. What tells the language service is to not to load anything from the project file.

[johnnyreilly]

@mhegazy,

Yes I am using Gulp to build.

The project file is kind of extraneous with this approach. I suppose I'd imagined that someone using Visual Studio would automatically be using a project file as they would in a non-TypeScript project. And if they were, I was keen to see if there was a way to use the project file without opting into the implicit file referencing. I think we've established this is possible with <TypeScriptEnabled>false</TypeScriptEnabled>?

Though you raise an interesting point - I'm guessing you're thinking that if the user is using Visual Studio just as an editor that they'd rely on the TypeScript Virtual Project instead? That way they can entirely bin the project file.

As you rightly say, compiler options are the other piece of the puzzle. From what you've said earlier, without a project file you have no control over compiler options. That's a shame - I'm a --noImplicitAny kind of chap and I'd certainly miss that.

From what I understand, a cross IDE project using a common approach is pretty much do-able in the following ways:

1. With <TypeScriptEnabled>false</TypeScriptEnabled> in a project file. This approach requires that all developers (regardless of IDE) use /// references to build context. Compiler options in VS can be controlled using the project file as is.
2. Using Visual Studio without any csproj tweaks. This approach requires that all /// references are maintained to build context outside of Visual Studio. It's possible that /// references and the csproj could get out of line - care required to avoid this. Compiler options in VS can be controlled using the project file as is.
3. Using just files in Visual Studio with /// references to build context. This scenario also requires that all developers (regardless of IDE) use /// references to build context. In Visual Studio there will be no control over compiler options. 😢

Are there any plans to allow the control of "default" TypeScript compiler options in Visual Studio in the absence of a project file? This would help with approach 3.

I think approach 1 might be the best cross-IDE approach available at present in my view.

[mhegazy]

@johnnyreilly i think you summed up nicely. thanks for the write up.

I do not see a problem with us exposing more options as global settings for loose files, e.g. --noImplicitAny, and --target. I have created #1223 to track that. but there are ones that we know we can not support globally, e.g. out, outDir, declarations. maproot, sourceRoot. There is room here for a proposal :)