Pluggable Compile-Time Module Resolution

Topics: Language Specification
Mar 17, 2014 at 12:37 AM

Hypothetical Compile-Time Plugins

Imagine for a moment that the Typescript compiler allowed simple plugins to be specified at compile time. These plugins are expected to conform to an API, and themselves have access to a simplified but stable compiler API. The plugins can alter the compilation process in subtle but meaningful ways. This has been publicly raised at least once before. The discussion of one possible type of plugin follows.

Pluggable Compile-Time Module Resolution

Module resolution occurs when the compiler encounters an import x = require('...'); statement. The (hypothetical plugin-enabled) Typescript compiler delegates module resolution to the plugin stack. Plugins can intercept the normal module resolution process, thus affecting compilation. Perhaps the standard compiler resolution process could itself be just another plugin within the stack?

Module Mapping Plugin

Let us assume that there exists a compiler plugin that maps module identifiers to physical files based on a set of external rules.

The plugin:
  • allows virtual module identifiers to be mapped to arbitrary physical files and directories (1),
  • allows the build-time module directory structure to differ from the run-time module directory structure,
  • allows .d.ts files for required external libraries to be located at arbitrary locations on the filesystem (1),
  • allows multiple root source directories to appear as a single rooted directory structure, with a selectable merge strategy.
(1) without the need to carefully place them at the Typescript expected location or the need to add /// <reference... or inline declare module '... statements

This plugin enables use-cases such as:
  • Better parity with RequireJS, which can be configured such that module 'paths' and identifiers are mapped to arbitrary file-system directories. This same flexibility should apply to Typescript source such that import x = require('...'); statements can be mapped to arbitrarily located Typescript files.
  • When writing NodeJS applications using Typescript, it would be convenient to have the definitions of external modules be automatically brought into scope simply through use of the require statement. The plugin would automatically locate the correct .d.ts file for that module based on information gleaned from the package.json file. No /// <reference ... would be needed. This Typescript compiler plugin might be but a small part of a larger toolset that would automatically download and manage the Typescript definitions for these external NodeJS modules. A similar use-case also exists for third-party browser AMD libraries.
  • In a large web application with multiple targets (many white-label instances, different mobile platforms, specific browser and OS flavours) it would be nice to be able to have more flexibility wrt source layout. Coupling this and the ability to specialise certain variants through the use of file-level overrides would offer a refreshing approach to managing the complexity of large multi-target applications.

RequireJS Text Plugin

Bringing textual resources into a module with a import template = require('text!./template.html'); statement is useful (especially combined with later RequireJS compilation/concatenation). The accompanying Typescript boiler-plate to accomplish these textual imports isn't so great. If a Typescript text plugin was loaded at compile-time, it could intercept all text! require statements and inform the compiler of the string type of these modules, resulting in zero boiler-plate.

Esoteric RequireJS Plugins

Here is a rather contrived (but useful nonetheless) example: a lazy! RequireJS plugin that effectively wraps an existing module in a lazy promise, which is loaded into memory/fetched only when thened. During RequireJS compilation, the modules comprising the main application would be concatenated into one JS file, and the ancillary modules (as specified with lazy!) would be automatically broken into other (concatenated) JS files. Such a mechanism would allow large applications to be segmented into chunks, allowing faster initial page-load times and deferred loading of lesser used functionality. This is possible already, but setting it all up is non-trivial. The role of the Typescript side of the puzzle here is quite small, the lazy! Typescript compile-time plugin would rewrite the type (only) of lazy! required modules so as to be wrapped in a promise type. Thus, this plugin 'decorates' the type of existing modules.

Flying Pig Example

Another example, this one a bit more dreamy (but one that I would love to see): a HTML-template to Typescript-module compiler. The compiler plugin could intercept all import template = require('typedTemplate!./some-template.html'); resolutions, and it could perform the template compilation on-the-fly. The resulting type of the template would reflect the shape of the view-model as loosely defined within the HTML. Why would anyone want strongly typed templates? The answer to that question closely matches the answer to 'Why Typescript?'. This discussion is getting off topic, but I can't resist saying that compile-time validation of the shape of the template 'scope' would be pretty awesome. The important point here is that these template dependencies would be auto-built as part of the normal Typescript compilation process, all enabled through the plugin capability. The dependency information is explicit and no extra pre/post build steps are required.

Potential Issues and Considerations

  • Plugins need to work within IDEs as well as from the command line.
  • Specifying the plugins and other Typescript options all on the command line would get tiring, and would differ from IDE to IDE, perhaps a standardised JSON configuration file might help?
  • Plugins need to work cross platform.
  • Plugins need to work within WSH (really?).
  • Defining and maintaining a stable API would require a lot of effort.
  • Plugin interactions would need to be carefully considered.

Summary

Personally, I think this is just the 'tip of the iceberg' of useful features that could arise from just having pluggable module resolution alone!

Plugins would allow the community to more easily innovate, and it could thus help to reduce the 'please add feature X' phenomenon that plagues most large monolithic applications.

But, not everything needs to be pluggable! Some things, like module resolution discussed above, seem simple. AST re-writing on the other hand, that seems like it could be much more complex. I don't know for sure, I'm just guessing. Other compilers could be reviewed to see how differing approaches to pluggability fair.

Some might argue that the additional functionality discussed here is possible right now by hacking the typescript source rather than using a plugin mechanism. While such arguments are truthful, they are not useful. The Typescript source is complex. For someone wanting to add but a small feature, it is not feasible to spend days (or weeks) understanding and becoming familiar with the internal workings of such a complex beast. Any hacks added by individuals would be tightly coupled to the internals of (that particular version of) the compiler. Combining these hacks would be problematic, thus excluding the mix and matching of additional functionality by the tool end-users. For most developers with other responsibilities (me at the least), the Typescript source is just too large to contemplate 'hacking'. It's not my full time job! Writing a small plugin on the other hand, that seems achievable (and very useful!).

In summary, a well thought out plugin API, guided and/or implemented by those who intimately know the inner design intricacies would really give Typescript the boost it deserves.

Other Ideas / Resources:

  • Source level Annotations/Attributes - and plugins that can process them at compile-time (DI anyone? Or what about auto-generated observable models from interface definitions? Etc, etc).
  • See this workitem for even cooler examples.