Rethink our VFS abstraction

[Veykril]

The current setup is brittle and tends to break very easily in unexpected ways (#18886 (comment) being an example). The system is also somewhat difficult to follow (the way we partition the VFS into sourceroots is not easily understandable). We should simplify this and clean it up. One idea we've had is to first and foremost move the VFS state into salsa (once we have migrated over to the new one), as currently we have a weird middle ground between the two (some info is outside salsa, some is inside). The other thing is that we need to likely add support for lazily loading file contents (which likely means incurring i/o within salsa queries).

[Veykril]

cc @davidbarsky opening this so we can keep track of it (might want to link to this from the perf tracking issue even)

[davidbarsky]

The other thing is that we need to likely add support for lazily loading file contents (which likely means incurring i/o within salsa queries).

Unless I misunderstood what you said, we can lazily load files and file contents in Salsa today. Regardless, my opinion is that we should try to batch IO as much as possible instead of being lazy.

cc @davidbarsky opening this so we can keep track of it (might want to link to this from the perf tracking issue even)

Yeah, I'll update that issue to include this.

[davidbarsky]

For posterity, I imagine that the new VFS will end up looking a lot like Dada's or Ruff's new Salsa-powered VFS abstractions.

[tamird]

Is #15836 also related?

[davidbarsky]

Yes, probably.

[alanz]

A note about source roots from our usage in ELP, which is based on RA.

We recently updated the project model to match internal buck2 targets. These generate a test target per test file, and each can have different config, which we store in AppData. To know which options to use for a file we need its AppData, which used to come via the source root. But, now having one source root containing one target does not hold. So we have to jump though hoops to get a mapping from FileId to application information, based on individual FileId, but falling back to source root.
See e.g. this commit