Build cache invalidation bug when creating new subprojects

[jwmerrill]

Bug Report

🔎 Search Terms

incremental build cache invalidation stale project

🕗 Version & Regression Information

• This changed between versions 3.3 and 3.4.

The problem is not applicable before typescript 3.0 because 3.0 introduced the --build flag.

Incremental compilation was introduced in TS 3.4, so it appears that this problem has existed for as long as incremental compilation has existed.

The problem still occurs as of Version 5.2.0-dev.20230602 (typescript@next at the time of writing).

⏯ Playground Link

This problem is not reproducible in playground because it is a build system problem.

I created a reduced test case repo to demonstrate this issue. It has 3 commits and 3 ts source files.

💻 Code

The initial project structure has 3 source files in two directories,

• src/main/const.ts
• src/main/index.ts
• src/hello/hello.ts

and a single sub-project

• src/main/tsconfig.json

The following commit adds a second subproject:

• src/hello/tsconfig.json

and moves the const.ts file from src/main/const.ts to src/hello/const.ts

🙁 Actual behavior

If you build the project (using tsc --build) at the second commit, then at the first commit, then at the second commit, tsc incorrectly reports an import error.

build/hello/hello.d.ts:1:23 - error TS2307: Cannot find module 'main/const' or its corresponding type declarations.

1 import { LogFn } from 'main/const';
                        ~~~~~~~~~~~~


Found 1 error.

These steps are described in more detail in the minimial reproduction README.

🙂 Expected behavior

The project builds without error.

After seeing the error, if I run

npx tsc --build --clean
npx tsc --build

the project builds without error.

Interpretation

The first build on the main branch creates a build/hello/tsconfig.tsbuildinfo cache file for the hello subproject.

The next build, on an earlier commit, updates build/hello/hello.d.ts to import from main/const (the original location of this file). This commit has no awareness of the hello subproject, and it does not touch build/hello/tsconfig.tsbuildinfo.

The final build, back on main, does not rebuild the hello subproject because it sees build/hello/tsconfig.tsbuildinfo and sees that the newest input of the hello subproject is older than this cache file. You can see this by running npx tsc --build --verbose

Output:

 > npx tsc --build --verbose
[12:44:45 PM] Projects in this build:
    * src/hello/tsconfig.json
    * src/main/tsconfig.json
    * tsconfig.json

[12:44:45 PM] Project 'src/hello/tsconfig.json' is up to date because newest input 'src/hello/const.ts' is older than output 'build/hello/tsconfig.tsbuildinfo'

[12:44:45 PM] Project 'src/main/tsconfig.json' is out of date because buildinfo file 'build/main/tsconfig.tsbuildinfo' indicates that some of the changes were not emitted

[12:44:45 PM] Building project '/Users/jason/src/tsc-build-cache-invalidation/src/main/tsconfig.json'...

build/hello/hello.d.ts:1:23 - error TS2307: Cannot find module 'main/const' or its corresponding type declarations.

1 import { LogFn } from 'main/const';
                        ~~~~~~~~~~~~


Found 1 error.

This is incorrect because one of the outputs of the hello project is stale, even though none of the inputs are newer than the build cache file.

Notes

This bug is a reduced version of a bug found by Desmos Studio in the process of migrating the Desmos Graphing Calculator to TS subprojects.

Our developers found that they were having to frequently clean the TS build directory when switching back and forth between branches that were either newer than or older than the migration to subprojects.

Related issues

• Bad cache when type checking project references #48981 might be reporting this same problem. That issue is currently marked "Needs More Info". This issue links to a minimal reproduction. The minimal reproduction does not have any unlisted transitive project references, so that is not the cause of this issue (but may be the cause of Bad cache when type checking project references #48981)

[sheetalkamat]

This is unfortunate but seems to working as intended per design limitation.

The issue with moving files between subProject without cleaning up the buildInfo is that buildInfo holds the hash of inputFile text and not output state which means changes that are present in output are not recognized.
When You switch to branch that doesnt contain hello subproject but a file in that structure, the file output for that file gets updated as expected but when you switch back to main branch and build, per tsconfig things are uptodate because input file text for hello.ts and rest of the files is same as what it was built with we wouldnt emit. (this functionality of not checking output hash is intentional to avoid having to read and store these hashes esp for big projects this cost adds up, also you could be minifying or transforming in place as a post process which might lead to file not matching). So in this step hellp.d.ts not being updated and having state from previous build is working per current design.

[jwmerrill]

I wonder if it could make sense to keep/check hashes of the .d.ts outputs for composite projects, even if you choose not to check hashes of js outputs.

Rationale:

1. .d.ts outputs are likely to be smaller than .js outputs, so checking them may be less costly
2. .d.ts seem less likely to be modified in place by other tools later in the pipeline
3. In --build mode with composite projects, the .d.ts outputs are used by the TS compiler itself for further type checking. If these files are not up to date, TS type checking is not correct.