with projectReferences:true, referenced source files are not watched

[starpit]

Expected Behaviour

With projectReferences set to true and webpack set up to watch (e.g. via webpack-dev-server), modifications to referenced source files should result in a webpack recompile.

Actual Behaviour

Modifications to the entry seem to result in a recompile, but referenced projects that might be dynamically imported do not seem to be watched.

Steps to Reproduce the Problem

the repository linked below has repro instructions

Location of a Minimal Repository that Demonstrates the Issue.

https://github.com/starpit/ts-loader-references-watch-bug

[johnnyreilly]

I suspect this is a duplicate of one of the existing project references issues. I believe @andrewbranch may be looking at this at some point, but I must confess I haven't been tracking this. If you or anyone else would like to assist it would be most welcome 🥰

[FrozenKiwi]

I'm stuck on the same issue. When we "yarn dev" start webpack-dev-server, the referenced project is built and compiled *.js files are written to disk. Subsequent changes trigger build updates, but these changes are not reflected in the compiled *.js files, and the consuming project does not update. Manually changing the compiled *.js files does trigger an update in the consuming project.

Behind this question appears to be a fundamental question about how project references should work? #815 (comment). I feel project references should get written to disk, as this would enable/take advantage of TSC incremental builds(?). The goal of workspaces/project references is (?) to have a seamless DX across your whole codebase - from core/utilities/components/app, while maintaining the referenced projects black-box-yness.

I think this is the issue being looked at in #1028, but I'm not sure. After several days (and nights) of determined digging to build the kind of DX I'm used to in native development, I feel like I know less about what is supposed to happen than when I started. In the meantime I've dropped down to using "tsc --build --watch" to compile and setting webpack to watch the output js files. This works, but then requires more setup for copying around css modules etc, which kinda seems to undo some of the magic of project references.

[FrozenKiwi]

(side note - I'd be happy to assist if I could get some guidance on what is -supposed- to happen. ts-loader is a substantial codebase, and I gave up digging when I realised that typescript & ts-loader both have file-watchers in them and I didn't know which of them should be responsible for tracking file updates).

[johnnyreilly]

Happy new year @andrewbranch! Is there any chance you could give @FrozenKiwi some pointers to get them started? I'm very keen to get people who are willing to help involved, but I'm not super knowledgeable about where the project references stuff has got to. It looks like @sheetalkamat started work on something but that that is paused at present?

[sheetalkamat]

@johnnyreilly I am waiting for help from @TheLarkInn to the road blockers I hit while working on project references PR #1028

[johnnyreilly]

Awesome - thanks for the update @sheetalkamat!

[OneCyrus]

@johnnyreilly I am waiting for help from @TheLarkInn to the road blockers I hit while working on project references PR #1028

what are the roadblocks there? maybe someone else can help. would be awesome to get this working.

[johnnyreilly]

what are the roadblocks there? maybe someone else can help. would be awesome to get this working.

I'm afraid I don't know; if @sheetalkamat or @TheLarkInn are able to advise that'd be amazing and could possibly help someone else move this forward.

As always I'd like to say, this is open source and there's no requirement on anybody to do anything. Everything that is done is appreciated; no problems if there's other things going on. ❤️🌻

[sheetalkamat]

Here is the progress and road block that I have hit. If anyone can help, I can work with that person to see what needs to be done.

The work in progress ts-loader changes for project references are at:: #1028

The steps to setup ts-loader are::
• git clone https://github.com/TypeStrong/ts-loader.git
• cd ts-loader
• git checkout packageJsonExists
• yarn install
• yarn build

The projects in test\comparison-tests\projectReference* are the examples for the project reference scenario.
There are two main categories that work and one that doesn’t. In all tests there is project lib inside them that is referenced by the outer project.

Working scenario

1. cd test\comparison-tests\projectReferences && webpack –config webpack.config.js –watch
   Here webpack asks for output of app.ts, as part of that we find that it depends on lib project so we build it on disk as if tsc -b was called on it.. Then we let webpack know that app.ts depends on lib/index.ts. Since .ts is file is next to .js file webpack picks .ts and asks for output of lib/index.ts and we read it from disk (or use cached version if it was generated when output of app.ts was asked). Webpack watches app.ts as well as lib/index.ts

2. cd test\comparison-tests\projectReferencesOutDir && webpack –config webpack.config.js –watch
   Here as usual webpack asks for output of app.ts and we build lib\index.ts as part of this and give the output of app.ts after that. We also add lib\index.ts as dependency of app.ts
   Webpack then asks for output of lib\index.ts since there is no lib\index.js and it needs to resolve this “./lib” reference in app.ts/.js (that’s my guess) (note the output for lib\index.ts is in “lib\out\index.js” and then webpack watches lib\index.ts and edits do restart compilation.

Scenario that is not working

1. cd test\comparison-tests\projectReferencesOutDirWithPackageJson && webpack –config webpack.config.js –watch
   here when webpack asks for output for app.ts we build and generate outout lib\out\index.js as part of this step and also tell webpack using addDependency that app.ts also depends on lib\index.ts
   Webpack resolves the “./lib” reference in the app.js to lib\out\index.js because there is package.json pointing to it and it never asks for emit of “lib\index.ts” and changes to it never start new compilation

I think in general we want to be able to convey that lib\out\index.js depends on lib\index.ts since that’s just simple case if there are transitive references (eg. lib\index.ts depending on lib2\index.ts) we want to specify that as well. Typescript knows the output and input filename relation for these and that is not being able to conveyed is causing this in my opinion.

[sokra]

I just read the typescript documentation about project references and never used it, so I may not be aware of all the details of project references.

That's how I understand ts-loader should work with projectReferences by the README:

• It's builds all referenced projects like with tsc --build
• The current project uses generated *.d.ts files for the referenced projects
• It does not generate any output files (*.js) for the current project. These are passed to webpack.
• It may not run ts-loader on files of the referenced project when they are not imported. e. g. when only types are imported.
  • In this case the referenced project is not watched as ts-loader never run there.
• It may run ts-loader on files of the referenced project when they are imported. This is like a separate scope.
  • In this case the referenced project is watched as `ts-loader runs there.
  • But as *.d.ts file is not generated in this case, some error may not pop up in the parent project as they should.

Correct me if this is wrong...

Note: It's pretty unusual for webpack loaders to write files to the disk. I heard some rumors that the typescript compiler doesn't allow to read files from memory, so I we need to tolerate that. Writing files into the current directory probably lead to weird repeated watching triggers, so you may check if it's possible to use a separate directory for output files (node_modules/.cache/ts-loader/... may be a good one) or make sure that files are not rewritten if the content is equal (to not trigger watch events). Otherwise watchOptions.ignored is a workaround.

Anyway here is a design I would propose:

But first some notes:

You can add and compile an unconnected module with the this.loadModule API from the loader context. This is like an import "..." but the module is not loaded and not added to any chunk. Instead you get the module source code as result from the API. When calling this.loadModule multiple times with the same request only a single module is created. The this.loadModule API returns once the module has been built and added (loaders in the request has been run).

So here my design idea:

When ts-loader runs on a .ts file with a tsconfig.json with references it calls this.loadModule with a request like ts-loader/project-reference-loader.js!/path/to/referenced/tsconfig.json. It waits until this.loadModule returns and continue as usual. It can assume that the referenced project has been build. It also adds fileDependencies (with this.addDependency) to to all files returned by the ts-loader/project-reference-loader.js passed via this.loadModule.

ts-loader/project-reference-loader.js is a separate loader which takes care of building referenced projects. The result will be a list of dependencies. It calls typescript in some way on the whole project and also uses this.loadModule with itself (ts-loader/project-reference-loader.js) for all project references of the project. If aggregates all dependencies of the project (from the typescript compiler) and all dependencies returned from calling this.loadModule on the referenced projects. It calls this.addDependency for each of them and returns them as stringified source code (it need to generate source code, may as // ${JSON.stringify(dependencies). the result from this.loadModule need to be parsed again).

This should result in the following workflow:

ts-loader runs on each directly imported source file and returns the source code to webpack.
ts-loader/project-reference-loader.js runs on each project referenced directly and transitively.

As ts-loader/project-reference-loader.js depend on all project files it will recompile when one of them has changed. This will trigger a build of the project and makes sure it's always up-to-date.

As ts-loader also depends on all project files of referenced projects it will recompile when one of them has changed. As it loads ts-loader/project-reference-loader.js, it will wait until the project reference has finish, before recompiling itself, which may result in new type checking errors.

When a application file has changed only ts-loader of these file is invalidated and will recompile. It will still call this.loadModule on ts-loader/project-reference-loader.js, but as this module was not invalidated it will load from cache (not running the loader). Only the root typescript compilation is triggered this way.

Hope this helps

---

There is a prepend feature for project references. The most similar equivalent would in my opinion to add import statements to the referenced project. But the feature doesn't make a lot of sense in a modular world.

[johnnyreilly]

Thanks so much @sokra - that's incredibly helpful!

[sheetalkamat]

@sokra thank you.. I will experiment with your proposal in our next milestone making sure that typescript API is sound and make any necessary changes and post back.. It would be sometime later next week or a week after that..

[mariusheine]

Would be great to get this solved. We ran into this issue in a monorepo with yarn workspaces where we have a vue frontend (generated by vue cli) with some shared packages.

On initial yarn serve everything works and ts loader is building the shared packages. On file changes inside the frontend src directory or inside the dist directory of the shared packages (e.g. by manually run tsc -b inside a shared package) rebuild the frontend app. But changes in the src directory of the shared packages aren't recognized.

I was able to detect file changes by adding a context dependecy to the compilation inside a webpack afterCompile hook, but it did not rebuild the shared package.

If you need a tiny repro repo let me know.

PS: Unfortunately i am relatively new to webpack and typescript, otherwise i would like to help. I was looking into the code of ts-loader and was debugging a bit but i don't know what exactly is happening there :D.

[FlorianWendelborn]

Has anyone found a viable workaround until this is solved? E.g. manually telling webpack-dev-server to watch those files?

[FrozenKiwi]

I am currently just directly compiling those projects in my webpack config.

      {
        test: /\.ts(x?)$/,
        include: path.join(projectRoot, "app"),
        use: options.tsLoaders,
      },
      {
        test: /\.ts(x?)$/,
        include: path.join(sharedRoot, "src"),
        loader: 'ts-loader',
        options: {
          instance: "shared",
            configFile: path.join(sharedRoot, 'tsconfig.json'),
            transpileOnly: true, // fork-ts-checker-webpack-plugin is used for type checking
            projectReferences: true,
            logLevel: 'info',
        },
      },
      {
        test: /\.ts(x?)$/,
        include: path.join(utilsRoot, "src"),
        loader: 'ts-loader',
        options: {
          instance: "utils",
            configFile: path.join(utilsRoot, 'tsconfig.json'),
            transpileOnly: true, // fork-ts-checker-webpack-plugin is used for type checking
            projectReferences: true,
            logLevel: 'info',
        },
      },

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

Closing as stale. Please reopen if you'd like to work on this further.

[J-Rojas]

This is still a problem as of 2021. I'm using Webpack v4 and ts-loader v8.3.0. tsc --watch is able to monitor all project references. Is there any way ts-loader can hook into that to execute a build?