New function call overload resolution [0.9.5] makes impossible to describe typing for lot of extendable libraries

Topics: Language Specification
Dec 7, 2013 at 8:43 AM
I'm a contributor of DefinitelyTyped. I've got the error when was working on porting some definitions to new 0.9.5 version of TypeScript. It filed as #1976 then it was closed that it follows the spec.
That's true, the latest changes simplifies overload resolution.
But I found it very bad decision because it makes impossible to define many of existing JavaScript libraries.
Consider we have one library definition 'A.d.ts':
interface A {}

interface X {
    on(s: string, cb: (a: A) => any): X;
}
and we have another library definition 'B.d.ts', that extends A:
///<reference path="A.d.ts">

interface B {
    b: number;
}

interface X {
    on(s: "a", cb: (b: B) => any): X;
}
In the example no way to both refer the parent library and correct order more-specific overload.

It's reduced example of jQuery's extension.

The main reason that in some cases we can't order overloads correctly because definitions of the interface separated to files.
The one of the best benefit of using TS is the ability to describe existing JS libraries, with new simplifications (see other descussion) this feature comes less useful.
Dec 9, 2013 at 8:24 PM
I'd also like the TS team to reconsider this change, at least for specialization based on strings. It is very useful to be able to extend core functions, defined in e.g. lib.d.ts or the jquery specification, with custom overloads for a specific application. An example is custom jQuery extensions or custom events. These definitions are separated over different files, and trying to manage the order is sometimes even impossible (in the case of lib.d.ts).

I support simplification, but in the case of string overloads, I think 'specific strings first, then all other strings' is still pretty simple, and would solve a lot of use cases.
Developer
Dec 11, 2013 at 2:44 PM
I think there are some valid concerns raised here. It is indeed hard (and sometimes impossible) to arrange the order of definition files such that all specialized signatures appear first in merged types. The notion of giving higher priority to specialized signatures is one we discussed, but it gets tricky when signatures contain a mix of string literals and regular types. Consider:
function foo(x: string, y: string): Type1;
function foo(x: string, y: "world"): Type2;
function foo(x: "hello", y: string): Type3;
function foo(x: "hello", y: "world"): Type4;
Ideally, in the call foo("hello", "world") the rules would pick the last overload, so we would have to rank candidates first by number of matching string literals and then by order of declaration. It's possible to do, but it adds complexity. Ultimately we'll have to balance it against other things we'd like to do--including shipping 1.0 :-) But I hear your concerns.
Coordinator
Dec 12, 2013 at 12:11 AM
Just to follow up on Anders' comments. We had to balance some of the simplifications we were doing against some of the code patterns people wanted to use. One side effect is that the order of how you reference definition files now matters when you want to model plugins. If you'll allow me a contrived example:

plugin.d.ts
interface CentimetersResult extends Result {
    meters: number;
    centimeters: number;
}

interface Measure {
    (s: "cm"): CentimetersResult;
} 
baselib.d.ts
interface Result {
    value: number;
}

interface InchesResult extends Result {
    feet: number;
    inches: number;
}

interface Measure {
    (s: "inches"): InchesResult;
    (s: string): Result;
}
uselib.ts
///<reference path="plugin.d.ts"/>
///<reference path="baselib.d.ts"/>

var m: Measure;
var y = m("cm").centimeters;
The trick is that you need to reference the "plugin.d.ts" first, so its overloads are seen before the base library you're extending. As the definitions are merged together, you can use types from the base library in your extension, as I show. Once the .d.ts files are referenced in the correct order, you can then use the merged interface in the .ts file that consumes them.

It's a couple more steps to remember vs 0.9.1.1, but should be possible in some cases. As Anders points out, others are difficult or impossible to model. We're definitely keeping an eye on this use case (with luck we'll be able to improve on it in the future).
Dec 12, 2013 at 8:46 AM
@JonTurner,

Thanks for the simplified example. I wonder if you could use it to cover off a number of other scenarios that spring to mind?

For instance, let's say we were going to need plugin.d.ts and baselib.d.ts throughout our project. I'm guessing we couldn't rely on the implicitly referencing functionality anymore and we'd need to add in a _references.ts that stated the reference order required:
///<reference path="plugin.d.ts"/>
///<reference path="baselib.d.ts"/>
Would this work as I imagine?

On a separate topic I'm wondering how this scenario works for typing libraries in DefinitelyTyped. I'm wondering if plugin.d.ts would need to be have a reference added to it's header so that it references baselib.d.ts like this:
///<reference path="baselib.d.ts"/>

interface CentimetersResult extends Result {
    meters: number;
    centimeters: number;
}

interface Measure {
    (s: "cm"): CentimetersResult;
} 
Is that necessary? Will adding that reference in cause any problems when you are consuming plugin.d.ts elsewhere? Would your uselib.ts example still work with this tweak in place?

Alternatively is adding that reference unnecessary? Should we actually only be enforcing the reference order where we consume the library rather than directly stating a reference in plugin.d.ts as I have here?

Finally, you're no doubt aware that the DefinitelyTyped unit tests are essentially TypeScript files containing examples of how the library under test can be used. If a test file compiles it is considered to have passed the test. So we might have plugin-tests.ts file that looks like this: (we're essentially re-using the uselib.ts file to create our test file)
///<reference path="plugin.d.ts"/>
///<reference path="baselib.d.ts"/>

var m: Measure;
var y = m("cm").centimeters;
Does all this sound correct / hang together as I imagine? Apologies if my examples are somewhat stating the obvious but I just want to get a little clarity on the approach we should be using.
Coordinator
Dec 12, 2013 at 4:33 PM
Lots of good questions. Let me see if I can cover them.
For instance, let's say we were going to need plugin.d.ts and baselib.d.ts throughout our project. I'm guessing we couldn't rely on the implicitly
referencing functionality anymore and we'd need to add in a _references.ts that stated the reference order required
That's right. Inside of a VS project that uses implicit references, you would use the _references.ts file to order the .d.ts files so that plugins came first before the base libraries. The _references.ts file is specifically to handle times where order matters, like function resolution and controlling codegen order when using --out.
On a separate topic I'm wondering how this scenario works for typing libraries in DefinitelyTyped. I'm wondering if plugin.d.ts would need to be have a
reference added to it's header so that it references baselib.d.ts like this
One way I think about this is to think of plugin.d.ts as a "partial" that can't live by itself. That is, it's going to be up to the user to include it along with the proper dependencies, and you can't add references to it to make it more complete. If you add the ///<ref> as you show, your hunch is right - this will prevent the signature in plugin.d.ts from being seen, and once you make the change, you'll get an error in uselib.ts. While plugin.d.ts is a "partial" definition that requires the other baselib.d.ts, baselib.d.ts is standalone and can be used by itself.
Finally, you're no doubt aware that the DefinitelyTyped unit tests are essentially TypeScript files containing examples of how the library
under test can be used. If a test file compiles it is considered to have passed the test. So we might have plugin-tests.ts file that looks like
this: (we're essentially re-using the uselib.ts file to create our test file)
Exactly. The unit test can control the reference order, and verify that the specialized signatures are working properly. You could even go a step further and make sure that overloads across both references work correctly:
///<reference path="plugin.d.ts"/>
///<reference path="baselib.d.ts"/>

var m: Measure;
var a = m("cm").centimeters;
var b = m("inches").feet;
var c = m("none of the above").value;
Dec 13, 2013 at 9:06 AM
That's a fantastic explanation Jon - thank you! It might be good to share this explanation more widely. (I'm not sure how many people are watching this discussion)

For my part I'll try and let the Definitely Typed community know about this as people are working on porting typing libraries from 0.9.1 to 0.9.5.
Dec 13, 2013 at 10:06 AM
Edited Dec 13, 2013 at 10:14 AM
Thank you @ahejlsberg that you read our discussion and heard our concerns. I think ranking of overloads will be good decision and should't add any valuable performance cost.
Thank you @jonturner for some explanation and workarounds of the problem. But it still looks like some hack or even spike-nail =)

I think that ordering of the referenced files is not a good idea.

Child library must describe by itself the references that is used. It's a principle of responsibility.

This principle used everywhere in development: .Net assemblies describe referenced assemblies, Dlls have imports table, NuGet, node and other packages systems do it too. So the TypeScript will be inconsistent with this. More that this, it's inverse it wit such implementation.

Consumer of the libraries should not know how to order dependencies of the libraries. In other case, every typing definition provider that depends on another must also give the instructions how to order references.
Developers who use the definition won't even look at these instructions. If they provide invalid order, code will be working still, but they will lose more-specific more-useful more-safe type information.

Another point of inconsistent. We can use any of the types before it described. The code compiles well:
var a: A;
var n: number = a.g();

interface A {
    g(): number;
}
In this case we do not depend of the order of type declaration. We can also 'extend' interface in any place and it will take effect for whole code and we also don't depend of the order. So why we should take care of declaration order for function overloads?

PS. The last reason, I think, is also applicable for discussion 471751. I want to developers will take a look from my positions to.

It's a cry from the heart! =)
Dec 13, 2013 at 7:10 PM
Another TS enthusiasts & DefinitelyTyped contributor here:

+1 for Igorbek's case:

I too have problems with 0.9.5's ordering scheme: I'm used to doing a lot of patching and modification of definitions and it is now a harder to expand on existing overloads (it conflicted with my own patch-up hackery so much I stashed my projects 0.9.5 effort for now).

@Igorbek nicely described how easy it is to expand interfaces and types, and compared to this flexibility (and its creative freedom) the new overload ordering scheme is very strict and peculiar.

I feel it is limiting expression and usability of the typing platform if it become too fiddly to add or expand typing for existing code. We need maximum flexibility in how we work with multiple definition files (because practicalities are not always as tidy as we'd like).

If every user has to manually manage references and their complex ordering of all his sub/client modules then it becomes more of a drag to work typed.
Dec 20, 2013 at 8:59 AM
Edited Dec 20, 2013 at 9:00 AM
@ahejlsberg, @jonturner
I've found that you going to change behavior of interface merging in order to fix the problem discussed here.
That's definitely better then it was before. Thanks!

But I have still some doubts about your solution.
  1. I understand that situations when need to extend interface with less-specific overloads is uncommon, nonetheless could be. In this case we have mirrored situation.
  2. Overload resolution based on order can't decide situations described in other discussion.
Just think about this for future releases.
Dec 20, 2013 at 9:20 AM
Hi @ahejlsberg, @jonturner,

Just noticed the Interfaces now merge with later interfaces having a higher priority section under "Known breaking changes between 0.9.5 and 1.0".

It's not stated but I assume that this requires that enforce you interface order using _references.ts? I don't know how script ordering is determined in the absence of _references.ts?
Developer
Dec 21, 2013 at 12:54 AM
In Visual Studio script order is currently just the order that files appear in the Solution Explorer. So _references.ts is a means to more precisely control the script order in Visual Studio scenarios. Outside of Visual Studio script order is just the order that the files are passed to the compiler on the command line.
Jan 21, 2014 at 8:01 AM
Outside of Visual Studio script order is just the order that the files are passed to the compiler on the command line.
True. But note that you can just pass _references.ts to tsc outside of visual studio to get the same effect as visual studio
May 16, 2014 at 7:51 AM
Edited May 16, 2014 at 8:06 AM
I had the issue with the order of references, when I created node.js app and tried to use setTimeout function from node.js like so:
var timer: NodeTimer;
timer = setTimeout(somecallback, 1000);
The code didn't compile saying 'Cannot convert 'Number' to 'NodeTimer''
I understood that the compiler thought I was using standard setTimeout instead of the one from node.js

After posting a question on stackoverflow I was advised to create _references.ts file and explicitly state the order of references :
/// <reference path="Scripts/typings/node/node.d.ts" />
/// <reference path="C:\Program Files (x86)\Microsoft SDKs\TypeScript\1.0\lib.d.ts" />
This did solve the problem, though the user experience is really far from expected.