A suggested alternative to <reference> includes

Topics: General, Language Specification
Feb 17, 2013 at 8:57 PM
Edited Feb 19, 2013 at 1:23 PM
The reference include XML comment provides a mechanism for importing types from external definition or typescript files, e.g. /// <reference path="node.d.ts" />. The idea, while not being bad on paper (permits fine-grained control for importing types from various sources), is IMO one of the problem areas in TypeScript at present.

This level of fine-grained control creates a maintenance headache whenever files are renamed or deleted. It also doesn't help that the implementation itself is buggy, for example files with faulty references continue to compile. This generally induces a loss of confidence in the compiler.

The first step to fixing this is possibly to remove that fine-grained control. It should no longer be possible to reference individual files; rather, only references to other TypeScript projects should be permitted. References could be defined in a References.xml file per project. Here is an example.

Let's assume our solution contains three projects.

(1) C:\Trunk\Typings.proj
// File1: Node.d.ts
declare module "http" {
 export interface Server {
     listen(port: number): void;
 }
 export interface ServerResponse {
     end(data?: any, encoding?: string): void;
 }
 export function createServer(
    requestListener?: (request, response: ServerResponse) => void ): Server;
 }
// File2: Interfaces.d.ts
module my.interfaces {
    export interface IWeatherProvider {
        getForecast();
    }
}
(2) C:\Trunk\WeatherProvider.proj
// File: WeatherProvider.ts
module my.providers {
    export class WeatherProvider implements my.interfaces.IWeatherProvider {
        getForecast();
    }
}
(3) C:\Trunk\Foo.proj
// File: FooService.ts

/* Import examples  */
// * CommonJS or AMD
import http = module("http");

// * Import all types from module my.interfaces
import my.interfaces;

// * Create an alias for a specific type
import WeatherProvider =  my.providers.WeatherProvider;

http.createServer((request, response) => {
    var weatherProvider: IWeatherProvider = new WeatherProvider();
    response.end("The weather today is " + weatherProvider.getForecast());
}).listen(8093);

console.log("Foo Service at 8093");
Basically in the simple example above we create an HTTP server in file "FooService.ts" (in project "Foo.proj"). The HTTP server type needs access to the http module and the IWeatherProvider interface from one external project and the WeatherProvider type from another. The code file declares import statements at the top in order to use external definitions.

We gain access to these definitions by adding the following References.xml file to the Foo project.
// File: References.xml
<References>
    <Reference Include="Typings">
        <HintPath>C:\Trunk\</HintPath>
    </Reference>
    <Reference Include="WeatherProvider">
        <HintPath>C:\Trunk</HintPath>
    </Reference>
</References>
In summary, here is the structure for the three projects:
  • C:\Trunk\Typings.proj
    Node.d.ts
    Interfaces.d.ts
  • C:\Trunk\WeatherProvider.proj
    WeatherProvider.ts
  • C:\Trunk\Foo.proj
    References.xml
    FooService.ts
The key points and benefits are the following:
  • The compiler identifies each referenced project and builds up its type repository from every ".ts" and "d.ts" file in those projects.
  • Every file that requires a specific type must declare that intention via import statements at the head of the file.
  • The declaration syntax unifies the "using" directive with the "import" statement.
  • The Reference.xml will integrate with Visual Studio once the tooling is in place for adding references to other projects; it should also work outside Visual Studio without a problem.
  • This works particularly well for the case when TypeScript projects are complied into a single file, in which case the TypeScript compiler is not required to search the file system for constituent files in a referenced project.
I wonder whether this will work? And also if there are better alternatives?

Noel
Feb 18, 2013 at 3:57 PM
I do not like this because TypeScript is not Visual Studio dependent and many do not have "projects" as you see them. Looking at the compiler, you'll see that file references are actually somewhat of an afterthought and not baked into the language/parser. Referenced sources are resolved and parsed and the entire set of source is then type checked. Ideally something besides ad-hoc XML in a comment would be better, but I don't know what.
Feb 19, 2013 at 1:34 PM
@cretz,

The suggestion above is not dependent on Visual Studio. The terminology may have lead to that conclusion, but one may replace the term "project" with "folder" (i.e. a directory on the file system) in the above without changing the conclusion.

So, all that is required is to organise TypeScript files in a folder structure, inform the compiler where those folders are located via the References.xml config file, and add suitable import statements in the code files.

Will that work for you? If not what objections do you have?

(What happens with Visual Studio is the following: The References.xml becomes integrated with the project settings file; Visual Studio provides an "Add Reference" UI that then manages the embedded References.xml for you)
Feb 19, 2013 at 4:36 PM
Firstly, TypeScript doesn't have a config file or anywhere to store metadata and I would be hesitant to want one added. Secondly, a general References.xml means you expect to share common references, but this is not necessarily the case. It is much more flexible for files to specify their own dependencies. For example, take a project which mostly has shared code but a section for node and a section for the browser...depending upon which entry point file you send to tsc makes a difference on what is compiled. Even TypeScript itself has three builds (typescript, tsc and typescriptServices) which very likely resolve different imports.

I do like the idea of automatic inclusion folders but I struggle to think of a way to provide meta data to the tsc compiler in a uniform way. Maybe the best way is to provide a --include=DIR of some sort on the tsc call, but then you would have to manually configure this in your VS to have it resolve the includes from the path. In the meantime, I always keep some type of "component.ts" at the top of my source w/ only ///<references and have all other files that are part of said "component" only reference that file. Granted, I have run into slight issues w/ ordering, but haven't been able to define the problems enough to submit a bug report.
Feb 20, 2013 at 2:59 PM
@cretz, forgive me, but I found too many things to disagree with the last post :-)

It is much more flexible for files to specify their own dependencies
This is why we have the problem in the first place. Yes it is flexible - but no, it's not scalable and creates a maintenance problem. We currently have 282 TypeScript code files, and there have been people on the forum who have quoted higher numbers. File-level options and configuration are probably not the way forward.

For example, take a project which mostly has shared code but a section for node and a section for the browser
I would consider splitting them into separate projects.

Even TypeScript itself has three builds
I don't think it makes sense to use the argument that "because TypeScript itself does so and so", because most people in the real world are probably not writing compilers.

[How to] provide meta data to the tsc compiler in a uniform way
This is the raison d'etre for having a config file.

Maybe the best way is to provide a --include=DIR of some sort on the tsc call_
This cannot be said to be the best solution. One cannot be expected to change compiler options whenever the project organisation needs to change. It just makes things too brittle.
Feb 20, 2013 at 4:48 PM
  1. I understand the scalability issue. That's why I suggested including "directories", but please don't make me have an XML file called References.
  2. Acknowledged. Unfortunately the componentization/modularization of TypeScript projects I believe is an issue right now. I've even found myself having to generate ambient definition files for my other TypeScript modules as sort of an "interface" between them.
  3. It's the largest available TypeScript project I know and in the absence of best practices, I know no other resource.
  4. I could get behind an optional config file that maybe used JSON or something besides being specific to references and XML.
  5. Changing config file, changing Jake script seem similar to me.
I think we can agree that metadata needs to be provided to tsc for large projects. I definitely don't want References.xml. Maybe it could piggy-back on package.json, component.json, .bowerrc, or something. Since the language services in VS are now exposed to alteration and since references are resolved outside of parsing (ref TypeScript.getReferencedFiles) and passed in to the compiler (ref: TypeScript.TypeScriptCompiler.addSourceUnoit) it shouldn't be difficult for you to give your idea a try.
Feb 23, 2013 at 10:12 AM
I agree with not adding XML as a dependency. Already the current <reference> tag inside the comment seems like a hack since proper parsing would require a real XML parser which is currently not the case. The way I look at it, references are more a compiler directive and could be easily handled outside the source files.

What I nowadays do it just create a build file containing all the source files and use this build file from then on. So for example on Linux:

find ./src -name "*.ts" > build

and from then on I can just run:

tsc @build

You could also add external dependencies (like declaration files of other projects) to that build file and other items like the ECMAScript version. I guess for really big and complex projects you would need a real build tool like Jake or Grunt.
Feb 23, 2013 at 10:51 AM

Just put all references in a references.ts file and reference that from each file.

On 17 Feb 2013 21:57, "nabog" <notifications@codeplex.com> wrote:

From: nabog

The reference include XML comment provides a mechanism for importing types from external definition or typescript files, e.g. /// <reference path="node.d.ts" />. The idea, while not being bad on paper (permits fine-grained control for importing types from various sources), is IMO one of the problem areas in TypeScript at present.

This level of fine-grained control creates a maintenance headache whenever files are renamed or deleted. It also doesn't help that the implementation itself is buggy, for example files with faulty references continue to compile. This generally induces a loss of confidence in the compiler.

The first step to fixing this is possibly to remove that fine-grained control. It should no longer be possible to reference individual files; rather, only references to other TypeScript projects should be permitted. References could be defined in a References.xml file per project. Here is an example.

Let's assume our solution contains three projects.

(1) C:\Trunk\Typings.proj
// File1: Node.d.ts
declare module "http" {
 export interface Server {
     listen(port: number): void;
 }
 export interface ServerResponse {
     end(data?: any, encoding?: string): void;
 }
 export function createServer(
    requestListener?: (request, response: ServerResponse) => void ): Server;
 }
// File2: Interfaces.d.ts
module my.interfaces {
    export interface IWeatherProvider {
        getForecast();
    }
}
(2) C:\Trunk\WeatherProvider.proj
module my.providers {
    export class WeatherProvider implements my.interfaces.IWeatherProvider {
        getForecast();
    }
}
(3) C:\Trunk\Foo.proj
 // CommonJS or AMD
import http = module("http");
// Import all types from module my.interfaces
import my.interfaces;
// Create an alias for a specific type
import WeatherProvider =  my.providers.WeatherProvider;

http.createServer((request, response) => {
    var weatherProvider: IWeatherProvider = new WeatherProvider();
    response.end("The weather today is " + weatherProvider.getForecast());
}).listen(8093);

console.log("Foo Service at 8093");
Basically in the simple example above we create an HTTP server in file "FooService.ts" (in project "Foo.proj"). The HTTP server type needs access to the http module and the IWeatherProvider interface from one external project and the WeatherProvider type from another.

We gain access to these definitions by adding the following References.xml file to the Foo project.
// File: References.xml
<References>
    <Reference Include="Typings">
        <HintPath>C:\Trunk\</HintPath>
    </Reference>
    <Reference Include="WeatherProvider">
        <HintPath>C:\Trunk</HintPath>
    </Reference>
</References>
In summary, here is the project structure.
  • C:\Trunk\Typings.proj
    Node.d.ts
    Interfaces.d.ts
  • C:\Trunk\WeatherProvider.proj
    WeatherProvider.ts
  • C:\Trunk\Foo.proj
    References.xml
    FooService.ts
The key points and benefits are the following:
  • The compiler identifies each referenced project and builds up its type repository from every ".ts" and "d.ts" files in those projects.
  • Every file that requires a specific type must declare that intention via import statements at the head of the file.
  • The declaration syntax unifies the "using" directive with the "import" statement.
  • The Reference.xml will integrate with Visual Studio once the tooling is in place for adding references to other projects; it should also work outside Visual Studio without a problem.
  • This works particularly well for the case when TypeScript projects are complied into a single file, in which case the TypeScript compiler is not required to search the file system for constituent files in a referenced project.
I wonder whether this will work? And also if there are better alternatives?

Noel

Read the full discussion online.

To add a post to this discussion, reply to this email (typescript@discussions.codeplex.com)

To start a new discussion for this project, email typescript@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com

Feb 23, 2013 at 3:28 PM
markrendle wrote:
Just put all references in a references.ts file and reference that from each file.
That's what I do. I have a file called include.d.ts in the top level of my project that includes all of the files in the project. Then I just have a simple reference line at the beginning of each file to include that file. It's not quite as easy as an automatic config file, but it's still as simple as adding a line whenever you add a file.
Feb 24, 2013 at 1:42 PM
Just put all references in a references.ts file and reference that from each file.

Yes, that is what we do. Just to be clear here is what we do at present:
  1. We maintain a single "_references.d.ts" for every project (we have over twenty). This file only includes references to those files that contain types that should be visible outside the project.
  2. We also maintain a "Master" reference file that only includes project-level reference files, as described in (1).
  3. Finally, individual code files then reference the "Master" reference.d.ts.
Problems with this
  • There is no way to enforce this. One cannot prevent a rogue developer from adding a reference to any file in any project, directly in a code file.
  • Combining all references into a master file means that every single file, in every single project has access to every single type across all projects! This not only tends to clutter up the intellisense, but also makes understanding the interaction more difficult.
  • We want to be in a position where we are able to pick and chose which project references what. But in order to do that we need to start referencing individual projects (rather than the master reference). This leads to the following problem:
  • The path in the <reference> comment refers to an actual path in the file system. References to the physical structure of one's projects should not be found in code files, because it prevents renaming or moving projects around.
It shouldn't matter if one is working on two or twenty projects - this is a fundamental problem that will eventually catch up with most people.

Noel
Feb 26, 2013 at 9:13 AM
There is certainly lots of stuff to be desired when working with references, however this is not necessarily something that is missing in typescript, it might as well be the tooling that should support this better.

I can see that you could get far with maintaining a "<project>.d.ts" which would point to all files defining a "project"... Not to be confused with project in Visual Studio terms, a "project" here is merely a file containing a well-known list of references to other files, reason I use the term project is that in a Visual Studio environment, it might be natural that that list of files originates from what is within a VS project, but in other editors this can be very different.

This would allow you to referencing all items in that "project" quite swiftly... But maintaining those files becomes quite annoying, that is where the tooling comes in in my opinion.... (Alternative could be "<module>.d.ts" but if you define 2 modules in one file that becomes problematic)

So rather than changing how TS works, we need tools that help us managing references.

Since you seem to be using VS, think about how ReSharper works with Assemblies and Imports, Adding a reference to another type script project would allow you to start using the types in there, ones you have typed "other.MyType" and the reference is missing, obviously the tooling should provide you with a "add reference to other.MyType" action...

And when you begin to think about it, the "<project>.d.ts" might slowly become obsolete again, while it would mean less "clutter" at the top of a ts file, it shouln't be needed, because often if you begin to get way to many references at the top, something might indicate that you headed out of a bad path anyways... (good old high cohesion low coupling principle seem to begin to be broken)

So all in all... I don't think TypeScript needs to change here, I think the tooling needs to be enhanced even further...
That said, as a small improvement, maybe referencing wildcards wouldn't be overall terrible, but if the tool is there to maintain references, it becomes obsolete...
Feb 26, 2013 at 9:14 AM
Edited Feb 26, 2013 at 10:49 AM
Sigh... Dual posts possible through updates on Codeplex, thought that was an issue the entire web got over 10-15 years back >.< (Please delete this if anyone can)
Mar 15, 2013 at 2:37 PM
At least in Visual Studio automatically make the _references.ts to work exactly like _references.js !!!
Mar 18, 2013 at 2:27 PM
Edited Mar 18, 2013 at 2:33 PM
I use T4 template to generate References.ts file containing references to all .ts files (as well as References.ts from referened project). Its a little complicated since references need to be in specific order (base class must be declared before child class). And if i move some file, i still need to change reference to Reference.ts, since there is no "~/References.ts" that would point to root directory of project.

here is the template http://pastebin.com/dGtwtRFu



But this has to be solved somehow in language, its just crazy that you have to link everything like in c times. At least if refactoring worked on this.
Mar 18, 2013 at 2:46 PM
Kikaimaru wrote:
Its a little complicated since references need to be in specific order (base class must be declared before child class).
That's either not right, or I've been extremely lucky with my application so far.
Mar 18, 2013 at 8:02 PM
@markrendle,

Base classes are included in the correct order only if the -out flag is set (i.e. all compiled JavaScript files are combined into a single file).

If you are responsible for your own bundling then you do need to worry about that.
Mar 18, 2013 at 8:05 PM
Oh, that'd be it then. I'm using a Grunt task with the -out flag to do my compilation.
Mar 19, 2013 at 1:23 PM
Edited Mar 19, 2013 at 1:28 PM
I am using --out too, and it doesn't work. But it didnt happen to me in 0.8.1. So it may be a bug (it definitely happnes now in 0.8.3)
Apr 25, 2013 at 6:57 AM
Edited Apr 25, 2013 at 6:59 AM
Hello,

I would like to suggest the following scheme.

When we develop projects we have to deal with modularisation, and especially to match two concerns :
  • separation of concerns (which enable to apply consequently tle "loose coupling, tight consistence" principle)
  • reuse of previously done modules
Another aspect we have to deal with in a large project is the ability to develop the sofware with different teams... It conducts consequently to manage different projects in the solution (ms terminology).

The proposed solution aims to manage all these constaints.

In order to achieve this, the developper would have to do :
  • each file has to be part of one module and only one (mandatory)
  • the compiler compiles all the files sharing the same module name into one file (like the --out directive) called "<modulename>.js"
  • a module wanting to use another module has only to use the amd-like statement "import mymodule=module('<modulename>')"
The compiler will look into automatically all the referenced project inside the current project to find modules, and also inside the imported "<modulename>.js" files if no reference to the project is done.

To summurize :
  • a module may be shared into different files and different projects (possibly)
  • the compiler generates one file per module
  • a module uses another module by an import statement, but without having to define the name of the file which is imported ; it is the role of the compiler to do this.
best regards
Xavier