External module resolution logic

[vladima]

Problem

Current module resolution logic is roughly based on Node module loading logic however not all aspects of Node specific module loading were implemented. Also this approach does not really play well with scenarios like RequireJS\ES6 style module loading where resolution of relative files names is performed deterministically using the base url without needing the folder walk. Also current process does not allow user to specify extra locations for module resolution.

Proposal

Instead of using one hybrid way to resolve modules, have two implementations, one for out-of-browser workflows (i.e Node) and one for in-browser versions (ES6). These implementations should closely mimic its runtime counterparts to avoid runtime failures when design time module resolution succeeded and vice versa.

Node Resolution Algorithm

Resolution logic should use the following algorithm (originally taken from Modules all toghether):

require(X) from module at path Y

If exists ambient external module named X {
  return the ambient external module 
}
else if X begins with './' or '../' or it is rooted path {
  try LOAD_AS_FILE(Y + X, loadOnlyDts=false)
  try LOAD_AS_DIRECTORY(Y + X, loadOnlyDts=false)
}
else {
  LOAD_NODE_MODULES(X, dirname(Y))
}
THROW "not found"

function LOAD_AS_FILE(X, loadOnlyDts) {
  if loadOnlyDts then load X.d.ts 
  else { 
    if  X.ts is a file, load X.ts
    else if X.tsx is a file, load X.tsx
    else If X.d.ts is a file, load X.d.ts
  }
}

function LOAD_AS_DIRECTORY(X, loadOnlyDts) {
  If X/package.json is a file {
    Parse X/package.json, and look for "typings" field.
    if parsed json has field "typings": 
    let M = X + (json "typings" field)
    LOAD_AS_FILE(M, loadOnlyDts).
  }
  LOAD_AS_FILE(X/index, loadOnlyDts)
}

function LOAD_NODE_MODULES(X, START) {
  let DIRS=NODE_MODULES_PATHS(START)
  for each DIR in DIRS {
    LOAD_AS_FILE(DIR/X, loadOnlyDts=true)
    LOAD_AS_DIRECTORY(DIR/X, loadOnlyDts=true)
  }
}

function NODE_MODULES_PATHS(START) {
  let PARTS = path split(START)
  let I = count of PARTS - 1
  let DIRS = []
  while I >= 0 {
    if PARTS[I] = "node_modules" CONTINUE
    DIR = path join(PARTS[0 .. I] + "node_modules")
    DIRS = DIRS + DIR
    let I = I - 1
  }
  return DIRS
}

RequireJS/ES6 module loader

• If module name starts with './' - then name is relative to the file that imports module or calls require.
• If module name is a relative path (i.e. 'a/b/c') - it is resolved using the base folder.

Base folder can be either specified explicitly via command line option or can be inferred:

• if compiler can uses 'tsconfig.json' to determine files and compilation options then location of 'tsconfig.json' is the base folder
• otherwise base folder is common subpath for all explicitly provided files

Path mappings can be used to customize module resolution process. In 'package.json' these mappings can be represented as JSON object with a following structure:

  {
    "*.ts":"project/ts/*.ts",
    "annotations": "/common/core/annotations"
  }

Property name represents a pattern that might contain zero or one asterisk (which acts as a capture group). Property value represents a substitution that might contain zero or one asterisk - here it marks the location where captured content will be spliced. For example mapping above for a path 'assert.ts' will produce a string 'project/ts/assert.ts'. Effectively this logic is the same with the implementation of locate function in System.js.

With path mappings in mind module resolution can be described as:

for (var path in [relative_path, relative_path + '.ts', relative_path + "d.ts"]) {
    var mappedPath = apply_path_mapping(path);
    var candidatePath = isPathRooted(mappedPath) ? mappedPath : combine(baseFolder, mappedPath);
    if (fileExists(candidatePath)) {
        return candidatePath
    }
}
return undefined

With path mappings it becomes trivial to resolve some module names to files located on network share or some location on the disk outside the project folder.

{
    "*.ts": "project/scripts/*.ts",
    "shared/*": "q:/shared/*.ts"
}

Using this mapping relative path 'shared/core' will be mapped to absolute path 'q:/shared/core.ts'.

We can apply the same resolution rules for both modules and tripleslash references though for the latter onces its is not strictly necessary since they do not implact runtime in any way.

[vladima]

@mprobst and @jrieken - can you please check if this proposal covers your scenarios?

[danquirk]

Need to cross reference with comments here too #247

[mprobst]

One question:
"2. If X begins with './' or '/' or '../' [...]"

What does it mean when X is e.g. '/foo/bar' and you calculate Y + X? Do you resolve X against Y, e.g. like a browser would resolve a URL, so that '/foo/bar' would result in an absolute path?

Regarding the path mapping, this approach would not quite solve our problem. What I want to express is that a source file should first be searched in location A, then in location B, then C, and so on. That should be true for all source files, not just for a specific subset (as you do with the pattern matching). The source code of the including file should not care where the included file is located.

I presume the harder problem is establishing what the path that is searched in those locations is exactly. If a file is loaded relative to a base URL, we could first resolve all paths relative to that file's relative URL to the base, and then use the result to look up in the search paths.

Given a file Y, a require('X'):

1. let path be resolve(Y, X). Question here: default to relative paths or default to absolutes?
2. for each include path i (passed on command line, current working directory is the implicit first entry)
   1. let effective path be resolve(i, path)
   2. if effective path exists, return effective path
   3. continue
3. throw not found.

The initially passed 'Y' from the command line would also be resolved against the include/search paths.

For example, for a file lib/a.ts and running tsc -I first/path -I /second/path lib/a.ts in a path /c/w/d, where a.ts contains a 'require("other/b")', the locations searched for b would be, assuming default to absolute paths:

1. /c/w/d/other/b.ts (+.d.ts, +maybe /index.ts and /index.d.ts)
2. /c/w/d/first/path/other/b.ts
3. /second/path/other/b.ts

If you specified ./other/b, the locations searched would be:

1. /c/w/d/lib/other/b.ts (+.d.ts, +maybe /index.ts and /index.d.ts)
2. /c/w/d/first/path/lib/other/b.ts
3. /second/path/lib/other/b.ts

This would allow us to "overlay" the working directory of the user over an arbitrary number of include paths. I think this is essentially the same as e.g. C++ -I works, Java's classpath, how the Python system search path works, Ruby's $LOAD_PATH etc.

[mprobst]

... oh and obviously, I mean this as a suggestion to be incorporated into your more complete design that also handles node modules etc.

[vladima]

What does it mean when X is e.g. '/foo/bar'?

Yes, module name that starts with '/' is an absolute path to the file

I think path mappings can solve the problem if we allow one entry of it to be mapped to the set of locations

{
    "*": [ "first/path/*", "/second/path/*" ] 
}

Having this update module resolution process will look like:

var moduleName;
if (moduleName.startsWith(../) || moduleName.startsWith('../')) {
    // module name is relative to the file that calls require
   return makeAbsolutePath(currentFilePath, moduleName); 
}
else {
    for(var path of [ moduleName, moduleName + '.ts', moduleName + '.d.ts']) {
        var mappedPaths = applyPathMapping(path);
        for(var mappedPath in mappedPaths) {
            var candidate = isPathRooted(mappedPath) ? mappedPath : makeAbsolute(baseFolder, mappedPath);
            if (fileExists(candidate)) {
                  return candidate;
            }
        }
    }
}
throw PathNotFound;

Note:
I do see a certain value of having a path mappings, since it allows with a reasonably low cost easily express things like: part of files that I'm using are outside of my repository so I'd like to load the from some another location. However if it turns out that all use-cases that we have involve remapping of all files in project and path mappings degrade to just include directories - then let's use include directories.

Out of curiosity, do you have many cases when code like require('../../module') should be resolved in include directories and not in relatively to file that contains this call?

[mprobst]

Re the code example, I think even relative paths should be resolved against the mapped paths. Imagine you have a part of your code base in a different physical location (repository), but still use the conceptually relative path. In general, I think it might be a good idea to have a logical level of paths that get resolved, and then those are matched against physical locations, but the two concepts are orthogonal otherwise - that is, you can have relative or absolute logical paths mapping to any physical location.

Our particular use case is that we currently exclusively use absolute rooted include paths (require('my/module') where my is resolved to the logical root of the source repository). Relative paths could be useful if you have deep directory structures, but would need to be clearly marked so that there's no ambiguity, e.g. by using ./relative/path.

I see that your example of path mappings is strictly more powerful, but at least from where I stand, I think include directories cover all we need, and might be simpler for tooling to understand & implement. YMMV.

[basarat]

Would be great if typescript.definition was supported. #2829

I am open to different suggestions if you want.