Is module confusion holding back TypeScript?

Topics: General, Language Specification
Jun 26, 2013 at 8:55 PM
Edited Jun 26, 2013 at 9:16 PM
TypeScript is a great leap forward and, IMHO, the best of breed when compared to other transpiled "altJS" languages like CoffeeScript, Traceur, Six, etc. I would love to see greater adoption, at least until ES6 becomes widely available, but it's still lagging far behind CoffeeScript (currently, 500 vs. 3500 related repos on GitHub).

One thing that I feel may be holding TypeScript back is general confusion around what the different module scenarios are, what compilation and post-compilation options are available and when they need to be used, and the conflicting module syntax required for each scenario and/or target environment.

I've been working exclusively on large TypeScript libraries and applications since TypeScript was first announced last year. It took me almost that long to fully understand things. Here's what I finally concluded...

I have identified 4 module scenarios below. The first two are directly supported by TypeScript. The second two are indirectly supported, requiring additional tools and/or boilerplate syntax.

Module scenarios:
  1. AMD
  2. CommonJS (e.g. Node)
  3. CommonJS for the browser
  4. UMD
AMD modules must use the --module AMD compiler option, use import module1 = require("./src/module1") syntax for loading dependencies, and typically export symbols from an "external" module definition. These modules can only be used in an environment that supports asynchronous module loading via the define function, e.g. in the browser when using a third-party AMD loader such as RequireJS.

CommonJS modules must use the --module CommonJS compiler option, also use import module = require("./src/module1") syntax for loading dependencies, and also typically export symbols from an "external" module definition. These modules can only be used in an environment that supports synchronous module loading via the require function, e.g. Node.

CommonJS for the browser (using require) is only possible if a third-party post-compilation tool, such as Browserify, is used to bundle the main module and all of its dependencies into a single JS file. Alternatively, you can use ///<reference/> syntax for dependencies, export global symbols from "internal" modules using module Namespace syntax, and then add the necessary script tag(s) to your HTML page, but that's not really AMD or CommonJS.

UMD (Universal Module Definition) modules can use either --module compiler option (since it doesn't matter) plus the --out compiler option to generate a single output file with some UMD boilerplate appended in order to target the appropriate AMD, CommonJS, or browser environment.

My point is this is all extremely confusing, not well understood by new or inexperienced TypeScript developers, and is not well explained or demonstrated in the TypeScript documentation.

What's needed:
  1. Comprehensive documentation and better code samples
  2. Eliminate the conflicting development requirements dictated by each scenario, e.g. import vs. ///<reference/> syntax and "internal" vs. "external" module definitions
  3. Provide "bundling" capability in the TypeScript compiler, similar to what Browserify does, to allow CommonJS modules to work in the browser out-of-the-box
As a start for #1, I have created a typescript-modules repository that contains minimal sample code targeting each of the four module scenarios described above. Perhaps developers will find this useful as a starting point for their own apps/libraries, or to better understand the currently confusing situation.

If #2 and #3 are addressed, developers should no longer have to worry about how to code modules up front. Only use the import syntax for loading dependencies, and simply let compiler options dictate the correct output for each scenario. I believe this will go a long way toward increasing TypeScript adoption and alleviating the confusion.
Jun 27, 2013 at 7:30 AM
You can also use CommonJS on the browser through use of 3rd party libraries which handle the file loading while the app is running - using XMLHttpRequests or similar.
Jun 27, 2013 at 4:17 PM
Edited Jun 27, 2013 at 4:18 PM
@Griffork, that's true. You could implement a require function that uses a synchronous XMLHttpRequest under its hood. However, I understand this has serious drawbacks including load performance and cross-domain requests. Dojo used to do exactly this, and it ended up being the main reason why Dojo 1.7 was such a major refactor and why RequireJS was developed.

But that's besides the point, which is the current module story in TypeScript is too confusing, and I think it would do well to simplify things. I want to be able to write my modules/libraries/apps in one way, and then let the TypeScript compiler build the correct output for my target environment (AMD/browser, CommonJS/node, CommonJS/browser, etc.)
Jun 30, 2013 at 5:22 PM
@billyzkid, yes the module story is confusing.

Just to be clear the second method that you mention under "CommonJS for the browser" goes like this:

Foo.ts
module foos {
    export class Foo {
    } 
}
Bar.ts
/// <reference path="foo.ts" />
module bars {
    
    export class Bar {
        public value = new foos.Foo(); 
    }
}
As you say this is neither AMD or CommonJS. Neither does it use Browserify (which only works on "require" calls). In short this is simply JavaScript code organised in separate files, with no dependencies on anything or any requirement to follow any conventions. This is the way we want it to be.

The files themselves are organised into separate TypeScript projects, so that we have the hierarchy: Global > Project > File.ts, where there is one global and many projects, each of which contains many files. The Global is the final JavaScript file that is saved to the deployment folder; from there it is served up by the web server, for example: "http://mydomain.com/scripts/global.js"

So how is Global.js generated?

We use a bundling mechanism that in debug mode splits Global.js into its constituent files, while in release mode it generates a single minified bundle. The splitting up helps with debugging, while with the released file the browser need only make a single request. (For those familiar with ASP.Net 4.0, this bundling mechanism shouldn't really come as a surprise.)

Now, I don't believe that TypeScript can or should get involved in the bundling mechanism, because TypeScript operates at the Project level while the bundling needs to occur at the Global level.

What we would like to see is better support at the Project level, in particular the ability to reference entire projects, and replacing the <reference> comment with import, but for namespaces, so that we may write:

module bars {

    import foos; // Foo.ts is in the same project if not we reference the project in which it exists
    
    export class Bar {
        public value = new Foo(); 
    }
}

I believe this is a common pattern for those developing for the web.

Noel
Jul 1, 2013 at 12:12 AM
@nabog
Doesn't that mean that global.js has to be re-bundled for each unit test?
Also, are you essentially saying that Typescript should cater for none of these, and should instead leave all implementation-specific features to a third party tool? Because I, for one, do not agree :).

I do however agree greatly with @billyzkid's suggestions, I would love to see bundling done in the compiler, and the compiler import optimisations brought back, (where an unused reference was optimised out).

Also I certainly agree that more documentation would be useful surrounding that issue, but I think that the documentation would probably be better done by third party sources (bloggers and such) rather than the Typescript team (since the more people who try to explain how something works, the more likely it is that people will find someone they can understand).
Jul 1, 2013 at 9:27 AM
@griffork,

The code files can be referenced individually for unit tests if necessary. If the unit tests are dependent on the framework within global.js then that is handled by the bundling mechanism. The ASP.Net bundler is not a bad implementation. It keeps track of the files that have changed and updates the bundle accordingly. For instance, we get sub second bundling performance over a code base of some 400 js files, when individual groups of files are rebuilt.

Just to rephrase my point above:
  • TypeScript should certainly consolidate the import syntax and do away with <reference> includes.
  • We also need to cater for the scenario I've sketched out above, which uses neither AMD or CommonJS, but where code is organised in Projects (I'm referring to Visual Studio projects, but one may think of them as folders on a file system). For this case, we need to be able to reference other projects and import namespaces from them.
This is looking beyond a small code base and at scalability: small website => A single project with multiple folders underneath. Larger website => Multiple projects, each with multiple folders underneath.

My point is that for TypeScript to effectively generate the bundle global.js it would need to span Projects - which it cannot do, because (in Visual Studio) each project is compiled individually. At what point can the bundling occur?
Coordinator
Jul 2, 2013 at 5:12 PM
@nabog - "TypeScript should certainly consolidate the import syntax and do away with <reference> includes."

We (and I) worked through a few design iterations to see if I could consolidate import and <reference> a few months back when the design was still in flux. Some of the things that came out of that:
  • Reference and import are trying to do two different things. With references you're saying "trust me, I'll make these available at runtime". With import, we're actually codegen'ing specific code for a style of module loading. These two things are dissimilar enough that it'd likely be confusing to try to come up with a hybrid.
  • Going along with the first point, consolidation would mean that users would be forced to code in a particular style. For example, if import was how we chose to consolidate, then users would have to learn module loading from the start and use it in every project. Having a simpler <reference> gives people an easy way into getting started and the option to only use loadable modules when it makes sense for the project.
I agree that it's extra cognitive load to have similar features. In the end, it's a trade-off between complexity, compatibility, and an easy way to get started.
Jul 3, 2013 at 10:37 AM
Edited Jul 3, 2013 at 10:38 AM
I would say that if imports got tweaked the other way would be unnecessary.

Actually right now you can use imports just as "declaration providers" without codegening anything. It just works...

Users would not have to learn anything new. Actually there is a bonus of not having to understand two different ways of inclusion.
In this case that kind of abstraction is already possible and very useful.

The only problem I see is the amount of code made at DefinititelyTyped which was written with ///<reference syntax in mind.
The reason was that imports weren't ready.. just as they still aren't fully ready.
Jul 7, 2013 at 11:13 AM
@jonturner,

yes, references and imports are doing two different things.

However, it shouldn't necessarily follow from that that nothing should be done about the problem.

TypeScript has fundamentally altered how we think about JavaScript. In particular, it is now possible to design fairly complex architectures for a JavaScript platform - thanks to the compile-time safety, and language enhancements offered by TypeScript.

We have a new landscape that is different from the old. We no longer think about writing JavaScript per se, but writing TypeScript.

So in this climate the argument that we hear from the people behind TypeScript that "there are certain JavaScript patterns that you can depend on" comes from a rather redundant view of the world. With regard to this discussion, the <reference> include mechanism is a glaring throwback to the JavaScript past. Conditional compilation directives such as #if(DEBUG) is another example where we need to think differently - we would like to see such constructs supported because the code base now demands it.

One possible solution to the <reference> problem is to permit a project level configuration file, possibly Config.json :
{
    "reference": "c:\projects\foo,  ..."
}
And since you point out that references and imports are not the same, a new keyword for importing types from the referenced projects:

module bars {

    using foos; // Okay if module foos exists in any .ts file under folder c:\projects\foo
    
    export class Bar {
        public value = new Foo(); 
    }
}

With something like this in place, we should be able to scale out the code base with more confidence.

TypeScript has enabled new things and we need new structures and tools to deal with them.
Jul 7, 2013 at 3:53 PM

In this vein, what I'd really like to see is a TypeScript project type, the output of which is a single (minified?) JavaScript file, on which a web project might take a dependency.

Jul 8, 2013 at 1:53 AM
Edited Jul 8, 2013 at 1:56 AM
Actually, I really like nabog's suggestion for a few reasons:
  1. Referenced files are available project-wide, which is a better representation of their behavior. At the moment the ///reference files are only "visible" in any module that includes them either directly or via an include chain.
  2. Using makes it extremely obvious which files are actually using the reference, as it's difficult to tell if a file is using a reference if it's included via an include chain.
  3. The references file can be supplied along with the compiled JavaScript with comments to indicate which files and which versions of those files are used in the project, which would be invaluable since you would no longer need to look at the source code to determine what files need to be included.
It would solve almost all of the problems with using definitelytyped and external libraries or setting up someone else's project to run on your server.

At compile time (if you're converting to JavaScript with intellisense support) you can convert usings to ///reference paths.

edit:
@nabog I presume your comment on the line using foos; is meant to refer to d.ts files?
Jul 19, 2013 at 3:17 PM
I could not agree more that the module subject is an obstacle for beginners. I introducted typescript in a new project and people almost rejected it because they got completly lost over modules, require, import and reference. So, I solved it by the codeing guideline: "dont use import or the --module flag ever"! This lowered the entrance learningcurve enormously.

The problem of loading, of course, remains.

What do other languages do?
  • c-languages: usually, there is an external tool (linker or dynamic linker) that links the dependencies together at compile time (or prepares them for dynamic linking)
  • java: the build-in classloader is conceptually similar to commonjs and require.js
  • c#: sorry, am not the right guy to answer that ;)
In javascript, everybody can do what he likes and standardization is hard and tedious. Which is why I dont believe in AMD nor CommonJS. There is already tons of legacy and proprietary stuff out there using their own approaches. Since references are quite similar to c-type includes, why not staying on that track? All that is needed is the following proposed tool:
  • For development mode: A tool, that checks at compiletime the dependencies declared as references (or using, I wouldnt care) and produces a loader script. This loader script is able to load all individual js files in an optimzed way (it could use requirejs for that). But the sources stay seperate and therefore are easy to debug.
  • For production mode: the v0.9 compiler's ability to pack everything into one file solves that already! However, problems start again if you want to build an application on several libraries. For that, the same approach like in dev-mode could be deployed.
Jul 19, 2013 at 8:13 PM
@sapix - it sounds like your team had zero experience with modules (amd, commonjs) outside of typescript anyway, so their experience is entirely understandable. Setting up RequireJS (for example) in a browser is a real PITA and the first time you do it, you think you're doing it right and you're likely not. Without solid in-the-field experience of revealing module patterns with pure javascript, adding this technique to the cognitive load of taking on a new language is too much for most people. Additionally, if you're targeting browsers, it's probably not a good comparison to bring up java/c++ etc.; the browser introduces some pretty unique concepts.
Jul 21, 2013 at 9:35 PM
Edited Jul 21, 2013 at 9:55 PM
@oisin:
Maybe I wasnt clear. I am not interested in using AMD and many people arent't. But they get confused nevertheless in typescript because of it. My point was to make a suggestion that would make "--module"-free typescript development complete.

And, btw, the comparisons look maybe a bit farfetchechd, but hey, doesnt java run in the browser and has features to load classes dynamically over the network at runtime? So, where exactly would you see the conceptual difference between the java classloader framework and requirejs again ;) ?
Aug 6, 2013 at 4:39 AM
Just getting into TS, and generally love it. So thanks to everyone who has expended energy to birth it. But this module thing is a sad state of affairs. Things should and can be much simpler.

Telling compiler where type information is should be done outside of source files (get rid of <reference>), and should optionally specify module name to deal with namespace clashes. A 'ts.refs' file with something like:

<moduleName>=<path/to/.ts> or
<moduleName>=<path/to/.d.ts>=<path/to/.js>

express=./d.ts/express.d.ts
./lib/logger.ts // default module name 'logger'
clashLogger=./3rdparty/logger.ts // override default module name 'logger' with 'clashLogger'
oldJs=./d.ts/oldJs.d.ts=./vendor/oldJs.js

then in .ts's

import express; // outputs "var express = require( 'express' );"
import ex = express; // outputs "var ex = require( 'express' );"
import logger; // outputs "var logger = require( './lib/logger' );"
import clashLogger; // outputs "var clashLogger = require( './3rdparty/logger' );"
import oldJs; // outputs "var oldJs = require( './vendor/oldJs.js' );" or " define( 'myMod', [ 'oldJs' ], funct... ) "