Skip to content

Prefer higher-priority inferences from generic signatures over low-priority inferences from the first pass #56939

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

Andarist
Copy link
Contributor

@Andarist Andarist commented Jan 3, 2024

fixes #56931

At the moment, I'm treating this as an experiment and my primary goal is to get some potential feedback from the test suite run (cc @jakebailey). It's not meant to be reviewed yet - unless somebody feels like it 😉 At the very least, I still have to adjust code comments.

@typescript-bot typescript-bot added the For Uncommitted Bug PR for untriaged, rejected, closed or missing bug label Jan 3, 2024
target[i] = source[i];
target[i].priority = InferencePriority.None;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO for myself:

this line might not be needed at all or the assigned priority should be the source priority without InferencePriority.ReturnType (the latter sounds like a good idea if those inferences should stay "upgradable"/mergeable). IIUC, no new inferences without InferencePriority.ReturnType can be made here anyway - all of those were already collected before the ones from generic signatures start to be included.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, yeah, None definitely doesn't feel correct, since None is actually the highest priority inference possible - and I don't think simply adopting source's inference for target when they overlap should guarantee this is the best inference possible across all potential inference sites.

But also - this is mutating the inference object, which you should avoid, since it means if that inference is reused in multiple contexts, you'll be mangling the priority in all of them (...we probably shouldn't be reusing an inference in multiple contexts, but there are probably some edge cases we decided it was OK in that this might break the invariants of). If we can't reuse the source inference wholesale (because its priority already doesn't contain ReturnType), then we should duplicate the source inference object with a new priority.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I initially picked None here because it's the most conservative thing to do. If I'm reasoning about it correctly. Right now (on main) we iterate over those generic signatures (conceptually at least):

  • if the inferences are overlapping then we don't try to use any of the new ones
  • if the inferences are not overlapping then we use the new ones but that means that when we overlap again we won't pick any new ones

So my conservative change adjusts this only slightly - it only allows a single "upgrade" when inferences are overlapping (and in such a case, we only update the overlapping one, this is important - we can't assign inferences that are not overlapping).

I thought that this, both, is just the smallest change that could be done here if this direction is promising at all and that there might be an argument to be made here that in this scenario the inferences made from earlier generic signatures should be preferred (left ones having the higher priority over the right ones). The latter doesn't match how "regular" inferences are gathered but it is how it implicitly works on main (when overlap happens the new inferences are ignored after all, so they can't be replaced/updated).

I could certainly experiment with relaxing this behavior and allow those upgrades continuously.


I pushed out a change using cloneInferenceInfo but I'm not sure if you meant that or cloneInferenceContext. And perhaps it should also be applied conditionally? but then that would imply that cloneInferenceInfo is the right choice here since ReturnType is relevant to the info and not to the whole context.

If we can't reuse the source inference wholesale (because its priority already doesn't contain ReturnType)

Does it mean that when the inference info of the source contains ReturnType then it's OK to reuse it and avoid cloning?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it mean that when the inference info of the source contains ReturnType then it's OK to reuse it and avoid cloning?

If you're not going to mutate it, you can reuse it. Probably.

I pushed out a change using cloneInferenceInfo

Yeah, that's the one.

I could certainly experiment with relaxing this behavior and allow those upgrades continuously.

Yeah, I just don't think an inference site like this is the end-all inference source within a signature context (eg, a direct inference w/o return type inference or any other priority modifiers) - there may be a better one elsewhere, and for that one to be preferred, this priority needs to not be None.

This is spitballing, but it may be better to think of this, rather than as "merging" the inference infos, instead as using information from the contextual signature inference process to make new inferences in the original context - and that inference should have an appropriate priority that might not be the highest available. Honestly, in that regard, all this cloning may be a bit misleading - if we're allowing overlapping inference results in the two lists of inferences, mergeInferences is only used here, and it may not be the clearest approach. It may be better to instead create a new inference context to do these return type/contextual parameter inferences within, and then do a inferTypes(context.inferences, getInferredType(newContext, i), context.inferences[i].typeParameter, newContext.inferences[i].priority & ~InferencePriority.ReturnType) for each type parameter at index i, which should better handle merging inferences of differing priorities together, without needing bespoke or duplicated priority merging logic. I dunno though, there may be more complexity with this approach.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to chew on this for a bit. As an additional data point - I think this issue is quite related to this code and perhaps could help guiding us to the best solution: #57072 (comment) .

One extra problem with allowing further inferences here is that context.inferredTypeParameters would get polluted more and more with the type parameters that might not get used or something. I still don't quite understand all of the implications of this algorithm though so maybe I'm missing something. I already have seen this happening today when experimenting with some stuff:

declare function compose<A, B, C>(f: (a: A) => B, g: (b: B) => C): (a: A) => C

declare function f<A, B>(a: A): B
declare function g<B, C>(a: B): C

const result = compose(f, g)
//        ^? const result: <A, B>(a: A) => unknown

Since those inferredTypeParameters are simply concatenated there is no good way of even rejecting the unused ones~ at some point.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since those inferredTypeParameters are simply concatenated there is no good way of even rejecting the unused ones~ at some point.

Yeah, at some point a filtering of that list to remove type parameters that definitely aren't used and logic for figuring out if the type parameter was actually used is probably needed for usability. It'll be a bit of a challenge, though, with all the deferred inferences we can make "maybe it's used later 🤷‍♂️" is probably a somewhat common case.

@Andarist Andarist force-pushed the fix/generic-signature-inference-over-low-priority branch from c50d57b to 31d0e8b Compare January 3, 2024 17:31
@RyanCavanaugh
Copy link
Member

@typescript-bot test this
@typescript-bot test top100
@typescript-bot user test this
@typescript-bot user test tsserver
@typescript-bot test tsserver top100
@typescript-bot run dt
@typescript-bot perf test this
@typescript-bot pack this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jan 3, 2024

Heya @RyanCavanaugh, I've started to run the parallelized Definitely Typed test suite on this PR at 31d0e8b. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jan 3, 2024

Heya @RyanCavanaugh, I've started to run the diff-based user code test suite (tsserver) on this PR at 31d0e8b. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jan 3, 2024

Heya @RyanCavanaugh, I've started to run the diff-based user code test suite on this PR at 31d0e8b. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jan 3, 2024

Heya @RyanCavanaugh, I've started to run the regular perf test suite on this PR at 31d0e8b. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jan 3, 2024

Heya @RyanCavanaugh, I've started to run the diff-based top-repos suite (tsserver) on this PR at 31d0e8b. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jan 3, 2024

Heya @RyanCavanaugh, I've started to run the diff-based top-repos suite on this PR at 31d0e8b. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jan 3, 2024

Heya @RyanCavanaugh, I've started to run the tarball bundle task on this PR at 31d0e8b. You can monitor the build here.

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jan 3, 2024

Hey @RyanCavanaugh, I've packed this into an installable tgz. You can install it for testing by referencing it in your package.json like so:

{
    "devDependencies": {
        "typescript": "https://typescript.visualstudio.com/cf7ac146-d525-443c-b23c-0d58337efebc/_apis/build/builds/159229/artifacts?artifactName=tgz&fileId=0A653F4D0966E368F3A72BCA2EAB641CCDB6450249F2735BB315BF1598B0E9F802&fileName=/typescript-5.4.0-insiders.20240103.tgz"
    }
}

and then running npm install.


There is also a playground for this build and an npm module you can use via "typescript": "npm:@typescript-deploys/[email protected]".;

@typescript-bot
Copy link
Collaborator

@RyanCavanaugh Here are the results of running the user test suite comparing main and refs/pull/56939/merge:

Everything looks good!

@typescript-bot
Copy link
Collaborator

@RyanCavanaugh Here are the results of running the user test suite comparing main and refs/pull/56939/merge:

There were infrastructure failures potentially unrelated to your change:

  • 1 instance of "Package install failed"

Otherwise...

Something interesting changed - please have a look.

Details

puppeteer

packages/browsers/test/src/tsconfig.json

@typescript-bot
Copy link
Collaborator

@RyanCavanaugh
The results of the perf run you requested are in!

Here they are:

Compiler

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 295,465k (± 0.02%) 295,385k (± 0.01%) -81k (- 0.03%) 295,348k 295,459k p=0.020 n=6
Parse Time 2.65s (± 0.31%) 2.64s (± 0.20%) ~ 2.64s 2.65s p=0.523 n=6
Bind Time 0.82s (± 0.00%) 0.82s (± 0.50%) ~ 0.82s 0.83s p=0.405 n=6
Check Time 8.15s (± 0.26%) 8.13s (± 0.22%) ~ 8.11s 8.16s p=0.103 n=6
Emit Time 7.11s (± 0.29%) 7.09s (± 0.21%) ~ 7.07s 7.10s p=0.104 n=6
Total Time 18.73s (± 0.17%) 18.68s (± 0.13%) -0.04s (- 0.24%) 18.65s 18.72s p=0.036 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 192,466k (± 1.23%) 193,465k (± 1.58%) ~ 191,443k 197,481k p=0.810 n=6
Parse Time 1.35s (± 0.47%) 1.35s (± 1.15%) ~ 1.32s 1.36s p=0.485 n=6
Bind Time 0.72s (± 0.00%) 0.72s (± 0.57%) ~ 0.72s 0.73s p=0.405 n=6
Check Time 9.24s (± 0.26%) 9.25s (± 0.42%) ~ 9.21s 9.31s p=0.746 n=6
Emit Time 2.62s (± 0.82%) 2.61s (± 0.63%) ~ 2.60s 2.64s p=0.868 n=6
Total Time 13.93s (± 0.18%) 13.94s (± 0.31%) ~ 13.86s 13.98s p=0.376 n=6
Monaco - node (v18.15.0, x64)
Memory used 347,415k (± 0.01%) 347,393k (± 0.01%) ~ 347,367k 347,421k p=0.065 n=6
Parse Time 2.46s (± 0.48%) 2.46s (± 0.33%) ~ 2.45s 2.47s p=0.738 n=6
Bind Time 0.92s (± 0.56%) 0.93s (± 0.00%) +0.01s (+ 0.72%) 0.93s 0.93s p=0.025 n=6
Check Time 6.89s (± 0.26%) 6.85s (± 0.35%) -0.03s (- 0.46%) 6.81s 6.87s p=0.023 n=6
Emit Time 4.05s (± 0.34%) 4.06s (± 0.49%) ~ 4.03s 4.08s p=0.187 n=6
Total Time 14.32s (± 0.23%) 14.30s (± 0.21%) ~ 14.26s 14.34s p=0.419 n=6
TFS - node (v18.15.0, x64)
Memory used 302,732k (± 0.00%) 302,682k (± 0.00%) -50k (- 0.02%) 302,669k 302,700k p=0.005 n=6
Parse Time 1.99s (± 1.56%) 2.00s (± 0.49%) ~ 1.99s 2.01s p=0.935 n=6
Bind Time 1.01s (± 0.83%) 1.01s (± 1.02%) ~ 1.00s 1.02s p=0.923 n=6
Check Time 6.30s (± 0.39%) 6.29s (± 0.38%) ~ 6.26s 6.32s p=0.571 n=6
Emit Time 3.58s (± 0.35%) 3.58s (± 0.42%) ~ 3.56s 3.60s p=0.620 n=6
Total Time 12.88s (± 0.40%) 12.88s (± 0.17%) ~ 12.85s 12.91s p=0.936 n=6
material-ui - node (v18.15.0, x64)
Memory used 506,814k (± 0.00%) 506,836k (± 0.00%) ~ 506,822k 506,860k p=0.054 n=6
Parse Time 2.58s (± 0.53%) 2.59s (± 0.47%) ~ 2.57s 2.60s p=0.215 n=6
Bind Time 0.99s (± 0.99%) 0.99s (± 0.52%) ~ 0.99s 1.00s p=0.386 n=6
Check Time 16.94s (± 0.30%) 16.95s (± 0.38%) ~ 16.85s 17.03s p=0.629 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 20.50s (± 0.28%) 20.53s (± 0.35%) ~ 20.42s 20.63s p=0.468 n=6
xstate - node (v18.15.0, x64)
Memory used 512,854k (± 0.00%) 512,886k (± 0.01%) ~ 512,842k 512,967k p=0.296 n=6
Parse Time 3.27s (± 0.19%) 3.27s (± 0.30%) ~ 3.26s 3.28s p=0.733 n=6
Bind Time 1.53s (± 0.41%) 1.54s (± 0.34%) ~ 1.53s 1.54s p=0.091 n=6
Check Time 2.83s (± 0.69%) 2.82s (± 0.22%) ~ 2.81s 2.83s p=0.463 n=6
Emit Time 0.07s (± 5.69%) 0.07s (± 5.69%) ~ 0.07s 0.08s p=1.000 n=6
Total Time 7.71s (± 0.27%) 7.71s (± 0.30%) ~ 7.68s 7.75s p=0.627 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Angular - node (v18.15.0, x64)
  • Compiler-Unions - node (v18.15.0, x64)
  • Monaco - node (v18.15.0, x64)
  • TFS - node (v18.15.0, x64)
  • material-ui - node (v18.15.0, x64)
  • xstate - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,347ms (± 0.68%) 2,337ms (± 0.27%) ~ 2,327ms 2,345ms p=0.297 n=6
Req 2 - geterr 5,451ms (± 1.21%) 5,436ms (± 1.34%) ~ 5,376ms 5,547ms p=0.574 n=6
Req 3 - references 326ms (± 1.31%) 324ms (± 1.05%) ~ 320ms 329ms p=0.373 n=6
Req 4 - navto 275ms (± 1.18%) 276ms (± 1.43%) ~ 271ms 280ms p=0.742 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 89ms (± 5.32%) 89ms (± 5.86%) ~ 84ms 95ms p=1.000 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,480ms (± 0.96%) 2,503ms (± 0.66%) ~ 2,485ms 2,524ms p=0.093 n=6
Req 2 - geterr 4,117ms (± 1.59%) 4,112ms (± 1.54%) ~ 4,068ms 4,239ms p=0.936 n=6
Req 3 - references 337ms (± 1.37%) 338ms (± 1.22%) ~ 334ms 345ms p=0.681 n=6
Req 4 - navto 285ms (± 0.50%) 286ms (± 0.96%) ~ 284ms 291ms p=1.000 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 86ms (± 5.40%) 87ms (± 6.00%) ~ 77ms 91ms p=0.514 n=6
xstateTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,605ms (± 0.68%) 2,617ms (± 0.28%) ~ 2,609ms 2,627ms p=0.336 n=6
Req 2 - geterr 1,723ms (± 1.91%) 1,732ms (± 1.36%) ~ 1,694ms 1,768ms p=0.689 n=6
Req 3 - references 117ms (± 6.57%) 111ms (± 9.23%) ~ 101ms 123ms p=0.256 n=6
Req 4 - navto 365ms (± 0.54%) 365ms (± 0.79%) ~ 359ms 367ms p=0.934 n=6
Req 5 - completionInfo count 2,073 (± 0.00%) 2,073 (± 0.00%) ~ 2,073 2,073 p=1.000 n=6
Req 5 - completionInfo 310ms (± 1.43%) 311ms (± 1.38%) ~ 304ms 316ms p=0.809 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstateTSServer - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 153.20ms (± 0.21%) 153.11ms (± 0.20%) -0.09ms (- 0.06%) 151.96ms 156.81ms p=0.008 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 228.54ms (± 0.14%) 228.34ms (± 0.14%) -0.20ms (- 0.09%) 227.06ms 231.94ms p=0.000 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 230.69ms (± 0.18%) 230.62ms (± 0.20%) -0.06ms (- 0.03%) 228.97ms 236.80ms p=0.045 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 230.04ms (± 0.20%) 230.12ms (± 0.23%) ~ 228.41ms 244.85ms p=0.118 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@typescript-bot
Copy link
Collaborator

Hey @RyanCavanaugh, the results of running the DT tests are ready.
Everything looks the same!
You can check the log here.

@typescript-bot
Copy link
Collaborator

@RyanCavanaugh Here are the results of running the top-repos suite comparing main and refs/pull/56939/merge:

Everything looks good!

@typescript-bot
Copy link
Collaborator

@RyanCavanaugh Here are the results of running the top-repos suite comparing main and refs/pull/56939/merge:

Something interesting changed - please have a look.

Details

Server exited prematurely with code unknown and signal SIGABRT

Server exited prematurely with code unknown and signal SIGABRT

Affected repos

backstage/backstage Raw error text: RepoResults8/backstage.backstage.rawError.txt in the artifact folder

Last few requests

{"seq":3379,"type":"request","command":"definitionAndBoundSpan","arguments":{"file":"@PROJECT_ROOT@/packages/integration/src/azure/AzureUrl.ts","line":41,"offset":35}}
{"seq":3380,"type":"request","command":"completionInfo","arguments":{"file":"@PROJECT_ROOT@/packages/integration/src/azure/AzureUrl.ts","line":46,"offset":18,"includeExternalModuleExports":false,"triggerKind":1}}
{"seq":3381,"type":"request","command":"completionEntryDetails","arguments":{"file":"@PROJECT_ROOT@/packages/integration/src/azure/AzureUrl.ts","line":46,"offset":18,"entryNames":["arguments"]}}
{"seq":3382,"type":"request","command":"references","arguments":{"file":"@PROJECT_ROOT@/packages/integration/src/azure/AzureUrl.ts","line":75,"offset":14}}

Repro steps

  1. git clone https://github.com/backstage/backstage --recurse-submodules
  2. In dir backstage, run git reset --hard 5ccf26657140f7fd141e0a483da5b001576c6e26
  3. Install packages (exact steps are below, but it might be easier to follow the repo readme)
    1. In dir backstage/microsite, run yarn install --no-immutable --mode=skip-build
    2. In dir backstage, run yarn install --no-immutable --mode=skip-build
    3. In dir backstage/storybook, run yarn install --no-immutable --mode=skip-build
  4. Back in the initial folder, download RepoResults8/backstage.backstage.replay.txt from the artifact folder
  5. npm install --no-save @typescript/server-replay
  6. npx tsreplay ./backstage ./backstage.backstage.replay.txt path/to/tsserver.js
  7. npx tsreplay --help to learn about helpful switches for debugging, logging, etc

target[i] = source[i];
target[i].priority = InferencePriority.None;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, yeah, None definitely doesn't feel correct, since None is actually the highest priority inference possible - and I don't think simply adopting source's inference for target when they overlap should guarantee this is the best inference possible across all potential inference sites.

But also - this is mutating the inference object, which you should avoid, since it means if that inference is reused in multiple contexts, you'll be mangling the priority in all of them (...we probably shouldn't be reusing an inference in multiple contexts, but there are probably some edge cases we decided it was OK in that this might break the invariants of). If we can't reuse the source inference wholesale (because its priority already doesn't contain ReturnType), then we should duplicate the source inference object with a new priority.

@typescript-bot typescript-bot added For Backlog Bug PRs that fix a backlog bug and removed For Uncommitted Bug PR for untriaged, rejected, closed or missing bug labels Jan 10, 2024
@fatcerberus
Copy link

None is the highest priority inference

🎶 Everything you know is wrong
Black is white, up is down and short is long
And everything you thought was just so important doesn't matter
Everything you know is wrong
Just forget the words and sing along
All you need to understand is
Everything you know is wrong 🎶

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
For Backlog Bug PRs that fix a backlog bug
Projects
Status: Waiting on author
Development

Successfully merging this pull request may close these issues.

Low priority inference trumps a much better inference that can be made from a generic signature
5 participants