Merge pull request #935 from sheetalkamat/solutionBuilder

Using Solution builder to build project references

Author: andrewbranch

CHANGELOG.md

@@ -1,5 +1,8 @@
 # Changelog
 
+## v6.1.0
+* [Build upstream project references with SolutionBuilder](https://github.com/TypeStrong/ts-loader/pull/935) (#851, #913) - thanks @sheetalkamat!
+
 ## v6.0.4
 * [Fix issue when handling files not included in tsconfig.json](https://github.com/TypeStrong/ts-loader/issues/943) (#934) - thanks @davazp!
 

README.md

@@ -569,33 +569,7 @@ This flag enables caching for some FS-functions like `fileExists`, `realpath` an
 
 #### projectReferences _(boolean) (default=false)_
 
-**TL;DR:** Using project references currently requires building referenced projects outside of ts-loader. We don’t want to keep it that way, but we’re releasing what we’ve got now. To try it out, you’ll need to pass `projectReferences: true` to `loaderOptions`. You’ll also probably need to use TypeScript 3.1.1 or later (which, as of this writing, means `typescript@next`).
-
-ts-loader has partial support for [project references](https://www.typescriptlang.org/docs/handbook/project-references.html) in that it will _load_ dependent composite projects that are already built, but will not currently _build/rebuild_ those upstream projects. The best way to explain exactly what this means is through an example. Say you have a project with a project reference pointing to the `lib/` directory:
-
-```
-tsconfig.json
-app.ts
-lib/
-  tsconfig.json
-  niftyUtil.ts
-```
-
-And we’ll assume that the root `tsconfig.json` has `{ "references": { "path": "lib" } }`, which means that any import of a file that’s part of the `lib` sub-project is treated as a reference to another project, not just a reference to a TypeScript file. Before discussing how ts-loader handles this, it’s helpful to review at a really basic level what `tsc` itself does here. If you were to run `tsc` on this tiny example project, the build would fail with the error:
-
-```
-error TS6305: Output file 'lib/niftyUtil.d.ts' has not been built from source file 'lib/niftyUtil.ts'.
-```
-
-Using project references actually instructs `tsc` _not_ to build anything that’s part of another project from source, but rather to look for any `.d.ts` and `.js` files that have already been generated from a previous build. Since we’ve never built the project in `lib` before, those files don’t exist, so building the root project fails. Still just thinking about how `tsc` works, there are two options to make the build succeed: either run `tsc -p lib/tsconfig.json` _first_, or simply run `tsc --build`, which will figure out that `lib` hasn’t been built and build it first for you.
-
-Ok, so how is that relevant to ts-loader? Because the best way to think about what ts-loader does with project references is that it acts like `tsc`, but _not_ like `tsc --build`. If you run ts-loader on a project that’s using project references, and any upstream project hasn’t been built, you’ll get the exact same `error TS6305` that you would get with `tsc`. If you modify a source file in an upstream project and don’t rebuild that project, `ts-loader` won’t have any idea that you’ve changed anything—it will still be looking at the output from the last time you _built_ that file.
-
-**“Hey, don’t you think that sounds kind of useless and terrible?”** Well, sort of. You can consider it a work-in-progress. It’s true that on its own, as of today, ts-loader doesn’t have everything you need to take advantage of project references in webpack. In practice, though, _consuming_ upstream projects and _building_ upstream projects are somewhat separate concerns. Building them will likely come in a future release. For background, see the [original issue](https://github.com/TypeStrong/ts-loader/issues/815).
-
-**`outDir` Windows problemo.** At the moment, composite projects built using the [`outDir` compiler option](https://www.typescriptlang.org/docs/handbook/compiler-options.html) cannot be consumed using ts-loader on Windows. If you try to, ts-loader throws a "has not been built from source file" error.  We don't know why yet; it's possible there's a bug in `tsc`. It's more likely there's a bug in `ts-loader`. Hopefully it's going to get solved at some point. (Hey, maybe you're the one to solve it!) Either way, we didn't want to hold back from releasing. So if you're building on Windows then avoid building `composite` projects using `outDir`.
-
-**TypeScript version compatibility.** As a final caveat, [this commit to TypeScript](https://github.com/Microsoft/TypeScript/commit/d519e3f21ec36274726c44dab25c9eb48e34953f) is necessary for the `include` or `exclude` options of a project-referenced tsconfig file to work. It should be released in TypeScript 3.1.1 according to the tags. To use an earlier version of TypeScript, referenced project configuration files must specify `files`, and not `include`.
+ts-loader has opt-in support for [project references](https://www.typescriptlang.org/docs/handbook/project-references.html). With this configuration option enabled, ts-loader will incrementally rebuild upstream projects the same way `tsc --build` does. Otherwise, source files in referenced projects will be treated as if they’re part of the root project.
 
 ### Usage with webpack watch
 

package.json

@@ -1,6 +1,6 @@
 {
   "name": "ts-loader",
-  "version": "6.0.4",
+  "version": "6.1.0",
   "description": "TypeScript loader for webpack",
   "main": "index.js",
   "types": "dist/types/index.d.ts",
@@ -9,8 +9,9 @@
     "lint": "tslint --project \"./src\"",
     "comparison-tests": "tsc --project \"./test/comparison-tests\" && npm link ./test/comparison-tests/testLib && node test/comparison-tests/run-tests.js",
     "comparison-tests-generate": "node test/comparison-tests/stub-new-version.js",
-    "execution-tests": "node test/execution-tests/run-tests.js",
-    "test": "node test/run-tests.js",
+    "execution-tests": "git clean -xfd test/execution-tests && node test/execution-tests/run-tests.js",
+    "test": "git clean -xfd test/execution-tests && node test/run-tests.js",
+    "clean": "git clean -xfd test/execution-tests",
     "docker:build": "docker build -t ts-loader .",
     "postdocker:build": "docker run -it ts-loader yarn test"
   },
@@ -88,7 +89,7 @@
     "rimraf": "^2.6.2",
     "tslint": "^5.11.0",
     "tslint-config-prettier": "^1.15.0",
-    "typescript": "^3.5.2",
+    "typescript": "^3.6.2",
     "webpack": "^4.5.0",
     "webpack-cli": "^3.1.1"
   },

src/compilerSetup.ts

@@ -70,5 +70,13 @@ export function getCompilerOptions(
     compilerOptions.module = typescript.ModuleKind.CommonJS;
   }
 
+  if (configParseResult.options.configFile) {
+    Object.defineProperty(compilerOptions, 'configFile', {
+      enumerable: false,
+      writable: false,
+      value: configParseResult.options.configFile
+    });
+  }
+
   return compilerOptions;
 }

src/config.ts

@@ -4,6 +4,7 @@ import * as semver from 'semver';
 import * as typescript from 'typescript';
 import * as webpack from 'webpack';
 
+import { getCompilerOptions } from './compilerSetup';
 import { LoaderOptions, WebpackError } from './interfaces';
 import * as logger from './logger';
 import { formatErrors } from './utils';
@@ -149,3 +150,26 @@ export function getConfigParseResult(
 
   return configParseResult;
 }
+
+const extendedConfigCache = new Map() as typescript.Map<
+  typescript.ExtendedConfigCacheEntry
+>;
+export function getParsedCommandLine(
+  compiler: typeof typescript,
+  loaderOptions: LoaderOptions,
+  configFilePath: string
+): typescript.ParsedCommandLine | undefined {
+  const result = compiler.getParsedCommandLineOfConfigFile(
+    configFilePath,
+    loaderOptions.compilerOptions,
+    {
+      ...compiler.sys,
+      onUnRecoverableConfigFileDiagnostic: () => {} // tslint:disable-line no-empty
+    },
+    extendedConfigCache
+  );
+  if (result) {
+    result.options = getCompilerOptions(result);
+  }
+  return result;
+}

src/instances.ts

@@ -17,13 +17,18 @@ import {
   WebpackError
 } from './interfaces';
 import * as logger from './logger';
-import { makeServicesHost, makeWatchHost } from './servicesHost';
+import {
+  makeServicesHost,
+  makeSolutionBuilderHost,
+  makeWatchHost
+} from './servicesHost';
 import {
   appendSuffixesIfMatch,
   ensureProgram,
   formatErrors,
   isUsingProjectReferences,
-  makeError
+  makeError,
+  supportsSolutionBuild
 } from './utils';
 import { makeWatchRun } from './watch-run';
 
@@ -270,6 +275,23 @@ function successfulTypeScriptInstance(
     );
   }
 
+  if (configFilePath && supportsSolutionBuild(loaderOptions, compiler)) {
+    // Use solution builder
+    log.logInfo('Using SolutionBuilder api');
+    instance.configFilePath = configFilePath;
+    instance.solutionBuilderHost = makeSolutionBuilderHost(
+      scriptRegex,
+      log,
+      loader,
+      instance
+    );
+    instance.solutionBuilder = compiler.createSolutionBuilderWithWatch(
+      instance.solutionBuilderHost,
+      [configFilePath],
+      { verbose: true, watch: true }
+    );
+  }
+
   if (loaderOptions.experimentalWatchApi && compiler.createWatchProgram) {
     log.logInfo('Using watch api');
 

src/interfaces.ts

@@ -1,4 +1,4 @@
-export { ModuleResolutionHost } from 'typescript';
+export { ModuleResolutionHost, FormatDiagnosticsHost } from 'typescript';
 import * as typescript from 'typescript';
 
 import { Chalk } from 'chalk';
@@ -86,6 +86,14 @@ export interface TSInstance {
   program?: typescript.Program;
   hasUnaccountedModifiedFiles?: boolean;
   changedFilesList?: boolean;
+
+  solutionBuilderHost?: typescript.SolutionBuilderWithWatchHost<
+    typescript.EmitAndSemanticDiagnosticsBuilderProgram
+  >;
+  solutionBuilder?: typescript.SolutionBuilder<
+    typescript.EmitAndSemanticDiagnosticsBuilderProgram
+  >;
+  configFilePath?: string;
 }
 
 export interface LoaderOptionsCache {

src/servicesHost.ts

@@ -2,10 +2,12 @@ import * as path from 'path';
 import * as typescript from 'typescript';
 import * as webpack from 'webpack';
 
+import { getParsedCommandLine } from './config';
 import * as constants from './constants';
 import {
   CustomResolveModuleName,
   CustomResolveTypeReferenceDirective,
+  FormatDiagnosticsHost,
   ModuleResolutionHost,
   ResolvedModule,
   ResolveSync,
@@ -508,6 +510,83 @@ export function makeWatchHost(
   }
 }
 
+/**
+ * Create the TypeScript Watch host
+ */
+export function makeSolutionBuilderHost(
+  scriptRegex: RegExp,
+  log: logger.Logger,
+  loader: webpack.loader.LoaderContext,
+  instance: TSInstance
+) {
+  const {
+    compiler,
+    compilerOptions,
+    appendTsTsxSuffixesIfRequired,
+    loaderOptions: {
+      resolveModuleName: customResolveModuleName,
+      resolveTypeReferenceDirective: customResolveTypeReferenceDirective
+    }
+  } = instance;
+
+  // loader.context seems to work fine on Linux / Mac regardless causes problems for @types resolution on Windows for TypeScript < 2.3
+  const getCurrentDirectory = () => loader.context;
+  const formatDiagnosticHost: FormatDiagnosticsHost = {
+    getCurrentDirectory: compiler.sys.getCurrentDirectory,
+    getCanonicalFileName: compiler.sys.useCaseSensitiveFileNames
+      ? s => s
+      : s => s.toLowerCase(),
+    getNewLine: () => compiler.sys.newLine
+  };
+
+  const reportDiagnostic = (d: typescript.Diagnostic) =>
+    log.logError(compiler.formatDiagnostic(d, formatDiagnosticHost));
+  const reportSolutionBuilderStatus = (d: typescript.Diagnostic) =>
+    log.logInfo(compiler.formatDiagnostic(d, formatDiagnosticHost));
+  const reportWatchStatus = (
+    d: typescript.Diagnostic,
+    newLine: string,
+    _options: typescript.CompilerOptions
+  ) =>
+    log.logInfo(
+      `${compiler.flattenDiagnosticMessageText(
+        d.messageText,
+        compiler.sys.newLine
+      )}${newLine + newLine}`
+    );
+  const solutionBuilderHost = compiler.createSolutionBuilderWithWatchHost(
+    compiler.sys,
+    compiler.createEmitAndSemanticDiagnosticsBuilderProgram,
+    reportDiagnostic,
+    reportSolutionBuilderStatus,
+    reportWatchStatus
+  );
+  solutionBuilderHost.getCurrentDirectory = getCurrentDirectory;
+  solutionBuilderHost.trace = logData => log.logInfo(logData);
+  solutionBuilderHost.getParsedCommandLine = file =>
+    getParsedCommandLine(compiler, instance.loaderOptions, file);
+
+  // make a (sync) resolver that follows webpack's rules
+  const resolveSync = makeResolver(loader._compiler.options);
+  const resolvers = makeResolvers(
+    compiler,
+    compilerOptions,
+    solutionBuilderHost,
+    customResolveTypeReferenceDirective,
+    customResolveModuleName,
+    resolveSync,
+    appendTsTsxSuffixesIfRequired,
+    scriptRegex,
+    instance
+  );
+  // used for (/// <reference types="...">) see https://github.com/Realytics/fork-ts-checker-webpack-plugin/pull/250#issuecomment-485061329
+  solutionBuilderHost.resolveTypeReferenceDirectives =
+    resolvers.resolveTypeReferenceDirectives;
+  solutionBuilderHost.resolveModuleNames = resolvers.resolveModuleNames;
+
+  return solutionBuilderHost;
+}
+
 type ResolveTypeReferenceDirective = (
   directive: string,
   containingFile: string,

src/utils.ts

@@ -235,6 +235,9 @@ export function arrify<T>(val: T | T[]) {
 }
 
 export function ensureProgram(instance: TSInstance) {
+  if (instance.solutionBuilder) {
+    instance.solutionBuilder.buildReferences(instance.configFilePath!);
+  }
   if (instance && instance.watchHost) {
     if (instance.hasUnaccountedModifiedFiles) {
       if (instance.changedFilesList) {
@@ -351,6 +354,13 @@ export function validateSourceMapOncePerProject(
   }
 }
 
+export function supportsSolutionBuild(
+  loaderOptions: LoaderOptions,
+  compiler: typeof typescript
+) {
+  return !!loaderOptions.projectReferences && !!compiler.InvalidatedProjectKind;
+}
+
 /**
  * Gets the output JS file path for an input file governed by a composite project.
  * Pulls from the cache if it exists; computes and caches the result otherwise.

test/execution-tests/3.0.1_projectReferences/lib/tsconfig.tsbuildinfo

@@ -0,0 +1 @@
+{}

test/execution-tests/3.6.0_projectReferencesToBeBuilt/karma.conf.js

@@ -0,0 +1,17 @@
+/* eslint-disable no-var, strict */
+'use strict';
+var webpackConfig = require('./webpack.config.js');
+var makeKarmaConfig = require('../../karmaConfig');
+
+module.exports = function(config) {
+  config.set(
+    makeKarmaConfig({
+      config,
+      webpackConfig,
+      files: [
+        // This ensures we have the es6 shims in place from babel and then loads all the tests
+        'main.js'
+      ]
+    })
+  );
+};

test/execution-tests/3.6.0_projectReferencesToBeBuilt/lib/.gitignore

@@ -0,0 +1,4 @@
+*.d.ts
+*.js
+*.js.map
+*.tsbuildinfo

test/execution-tests/3.6.0_projectReferencesToBeBuilt/lib/index.ts

@@ -0,0 +1,5 @@
+export const lib = {
+  one: 1,
+  two: 2,
+  three: 3
+};

test/execution-tests/3.6.0_projectReferencesToBeBuilt/lib/tsconfig.json

@@ -0,0 +1,9 @@
+{
+  "compilerOptions": {
+    "composite": true,
+    "sourceMap": true
+  },
+  "files": [
+    "./index.ts"
+  ]
+}

test/execution-tests/3.6.0_projectReferencesToBeBuilt/main.js

@@ -0,0 +1,2 @@
+const testsContext = require.context('./', true, /\.tests\.ts(x?)$/);
+testsContext.keys().forEach(testsContext);

test/execution-tests/3.6.0_projectReferencesToBeBuilt/package.json

@@ -0,0 +1,10 @@
+{
+  "name": "basic",
+  "license": "MIT",
+  "version": "1.0.0",
+  "main": "index.js",
+  "devDependencies": {
+    "@types/jasmine": "^2.5.35",
+    "jasmine-core": "^2.3.4"
+  }
+}

test/execution-tests/3.6.0_projectReferencesToBeBuilt/src/app.ts

@@ -0,0 +1,5 @@
+import { lib } from '../lib';
+
+export function whatNumbersDoYouHave() {
+    return [lib.one, lib.two, lib.three];
+}

test/execution-tests/3.6.0_projectReferencesToBeBuilt/test/app.tests.ts

@@ -0,0 +1,7 @@
+import { whatNumbersDoYouHave } from "../src/app";
+
+describe("app", () => {
+  it("code compiles referenced project", () => {
+    expect(whatNumbersDoYouHave()).toEqual([1, 2, 3]);
+  });
+});

test/execution-tests/3.6.0_projectReferencesToBeBuilt/tsconfig.json

@@ -0,0 +1,11 @@
+{
+  "files": [
+    "./src/app.ts"
+  ],
+  "references": [
+    { "path": "./lib" }
+  ],
+  "compilerOptions": {
+    "noEmitOnError": true
+  }
+}