TypeScript: Better preprocessing to make importsNotUsedAsValues unnecessary

[dummdidumm]

Is your feature request related to a problem? Please describe.
Currently, TypeScript is processed on a per-file-basis. Because the script tag is only part of the truth, a transformer was written to re-add imports which to TypeScript look unused but are actually used in the template. This means that people need to be very careful about how to write their imports, which can be especially cumbersome if a type import is from the same file as a real import.

<script lang="ts">
  impor type { Type } from './foo';
  impor { Bar } from './foo';
</script>

Since TypeScript itself does not do that strict autocompletion separation (and they don't plan to change it, see this issue), people have to do it by hand.

So what would be ideal is to

• either somehow tell TypeScript which values are used so it does not throw them out, which would make the transformer unnecessary
• or write a more sophisticated transformer which traverses the AST and throws out all types from the imports (similar to Fix/remove type imports typescript #164, only without relying on a program if possible)

Describe the solution you'd like

Ideally, I don't have to be that strict with seperating type and value imports. I currently see two ways to get there.

Solution 1: The Svelte-AST-compile-route

This solution is similar to the one currently used in eslint-plugin-svelte3. Some logic of #155 could maybe be reused.

1. get the whole Svelte file, blank style content
2. transpile script tags' TS to JS using the current transform which preserves all imports
3. do a svelte.compile which will return a list of variables with the info where they are used (module, template, etc)
4. append all variables to the bottom of the TS code so TS sees this "fake usage" and does not throw out imports that are only used in the template
5. transpile TS to JS without the "keep all imports" transformer

Advantages:

• probably the most correct and safest varaint

Disadvantages:

• Now TS needs the whole markup and would technically be a markup, not a script transformer. That's likely a breaking change
• It wouldn't work if people use markup that needs preprocessing, or that would need to happen before the script transformation

Solution 2: Walk TS AST to find type only references

This solution would be similar to #164 . Basically, walk the AST and find all references of all imports, and filter those out who are type-only. Walking the AST without advanced capabilities like a type checker / TS program is likely tedious, but probably more robust.

Advantages:

• no breaking change, only needs script contents

Disadvantages:

• maybe not possible without expensive type checker (although solution 1 would also be slower)
• there may be edge cases where this does not work: If someone imports a type in module-script but uses it as a type in the instance-script. The logic would be "oh this is unused and not otherwise used as a type, preserve it" which would be wrong.

How important is this feature to you?
Important in order to boost TypeScript development productivity.

[kaisermann]

I also prefer solution 1.

Now TS needs the whole markup and would technically be a markup, not a script transformer. That's likely a breaking change

Not necessarily! We could have a way to access the whole markup of the current file in the script/style preprocessors (I'm not sure if we would have some race condition here). If the markup needs to be preprocessed, the internal markup preprocessor could update the shared "markup" content. This would mean that the standalone typescript preprocessor should also have a markup preprocessor.

Moving ts to the markup step would be not only a big endeavour, but we would also need a way to know if we're going to deal with TS or not, complicating things even more.

Edit:

That won't work. I forgot that Svelte runs all markup preprocessors first 😭

[dummdidumm]

What if we use this knowledge and tell people if they chain multiple preprocessors and they want the better DX they need to add svelte-preprocess as last in the chain? (This would mean this DX enhancement would be optional and opt-in) Then internally svelte-preprocess could put all results of previous markup preprocessors in some shared markup results map and use that. Maybe it's also worth a shot at talking to the other maintainer about enhancing the preprocess chain a bit by passing the complete current transpilation result (whole file content), which would remove the need for keeping such a map.

Solution 2 would not work I think, I thought about this some more and there could be the situation where an import is only used as a type in the script content but used as a value within the template, marking it a false positive for removal. So only a revival of #164 would work in all cases, which would mean slower compilation.

[SomaticIT]

Hello,

I'm investigating this issue because it's a real problem on big svelte/Typescript projects.

Maybe I can help on the implementation.

I did some tests on my local installation:

1. Collect whole file markup (after transformation) during the markup preprocessing phase and pass it to Typescript. It avoid moving Typescript preprocessor to markup but it's a little hacky.

2. Make svelte compiler pass the whole markup to when preprocessing (as a new option to avoid breaking change).

3. Move Typescript to markup (I don't like this).

My preference goes to solution 2 if it's not too complex. The simplest way (but not the cleanest) is solution 1.

Do you think it's possible to add this feature in svelte compiler?

The idea of appending variables used in markup seems the only viable solution to let Typescript knows which imports it can safely remove.

The problem is that it forces to compile ts and svelte twice. It could be problematic on large projects.

Is there a safe way to extract js expressions from markup without full svelte compile (maybe ast parser)?

If possible, it could allow us to avoid the double compilation and directly append expressions at the end of the script.

[axelthat]

Any updates?

[SomaticIT]

In my previous message, I ask two questions:

1. Is it possible to make the svelte compiler pass the entire markup to preprocess? (using an additional property to avoid breaking changes)

2. Is there a safe way to extract JS expressions from markup without doing a full svelte compilation?

If we can address this two questions, we can build a reasonable roadmap:

1. svelte: Allow svelte compiler to pass full markup to preprocessors
2. svelte-preprocess: Keep the actual process when no markup is available (to avoid breaking changes)
3. svelte-preprocess: Extract JS expressions from markup and append them to the bottom of the script
4. svelte-preprocess: Remove JS expressions from the built script
5. svelte-preprocess: Remove the custom import transformer

Note: I will try to investigate on question 1 and make a PR on svelte repository.

[dummdidumm]

I'm fairly certain question 1 can be answered with "yes"

[SomaticIT]

I'm too, that's why I will try to open a PR on this.

[SomaticIT]

I created a PR on the svelte compiler to allow it to pass the full source code to preprocessors.

This PR will allow us to avoid introducing the breaking change of making the typescript preprocessor a markup preprocessor as mentioned by @dummdidumm.

I'm still looking for a safe way to extract JS expressions from markup.

[SomaticIT]

I just found that if we remove the script and style tag then run svelte.parse to extract the AST, we can have the AST of the HTML markup. It seems to be a good start but there is a lot of cases to handle.

If think that we will need to walk the AST and generate a javascript/typescript equivalent of svelte expressions.

eg:

{#each list as item (item.id)}
    <p>{item.title}</p>
{/each}

should become:

for (const item of list) {
    let var0 = item.title;
}

What do you think?

I think that this solution allows to avoid the double compilation and allow to safely extract JS expressions from svelte.
The only problem I see is that we will need to be careful when svelte add new features. eg: NewBlock will be ignored until we implement it in the extraction process.

[SomaticIT]

Summary of the discussion from the svelte PR to avoid polluting the bad repo:

@dummdidumm wrote:
That sounds like too much work. The svelte compiler returns a list of variables referenced in the markup, which can be used instead

@SomaticIT wrote:
That sounds great. Do you know how can I retrieve this list?

If possible, I would like to avoid running the full compilation process since it will force us to do a double compilation and it will drastically impact the compiler performance

@SomaticIT wrote:
@dummdidumm, I tried to run the svelte compiler but the only way to make it reliable is to first compile typescript then compile svelte so the process will be very expensive...

Moreover, this way is not 100% reliable, even eslint-plugin-svelte3 is doing a full AST walk after compilation because some variables are not referenced: https://github.com/sveltejs/eslint-plugin-svelte3/blob/57ba6eca4af3df201be17d1fb7de80622ac7855e/src/preprocess.js#L147

Also, I suggest moving our discussion to the svelte-preprocess issue to avoid polluting this PR.
Do you agree?

Update: The above argument is erroneous as explained by @dummdidumm.

[dummdidumm]

Regarding the expensiveness: I don't think it is that expensive since it's a per-file-compilation, the Svelte compiler does not need to look at other files to compile the given file. Moreover, if you add generate: false it will skip creating JS/CSS code which should speed it up further. The eslint plugin code you linked is needed because you can create new variables through each/await/let blocks (like the item in {#each items as item}), which is irrelevant for us, so we should be good with vars. So what I envision is:

1. Strip scripts and styles from markup
2. Compile with the Svelte compiler
3. Extract vars which are referenced in the template from the result
4. For module-script: Join module-script, instance-script, referenced variables and transpile. For instance-script: Join instance-script and referenced variables and transpile.
5. Extract the transpile result which you care about (module-script or instance-script)