advice for high-performance language service usage

Topics: General
Nov 25, 2013 at 9:21 PM
We are developing the Eclipse plug-in and are finding that calling getDiagnostics() across all files in the workspace when a file is saved is somewhat slow (it takes ~7 seconds). I was wondering if there was any advice on how to do incremental diagnostic retrieval more efficiently. For example, is there a way to only check files which depend upon the changed file? Perhaps we aren't using the caches properly? etc...

In case you are curious, the code which calls into the language service is here:
https://github.com/palantir/eclipse-typescript/blob/master/Bridge/src/languageService.ts

Thanks!
Coordinator
Nov 25, 2013 at 11:40 PM
Getting all diagnostics kinds for all files is definitely an over kill, and will not scale with your project size. VS, for instance, uses some heuristics on when and for which files it calls getSemanticDiagnotics or getSyntacticDiagnostics. The host needs to be smart about when it queries for errors, when to invalidate errors, and what files to ask for errors for.
Nov 25, 2013 at 11:50 PM
Ah, that's a really good point - we're almost certainly calling getSyntacticDiagnostics too frequently since it should only be necessary for a file that was modified. My assumption is that getSemanticDiagnostics needs to be called whenever a file within a project is modified (unless references are tracked somehow) since a change to an exported type could break in other files. Are there any mechanisms you could suggest for making the semantic diagnostic checks more efficient?
Coordinator
Nov 25, 2013 at 11:59 PM
There is not much the compiler can do about it, the work here needs to be done on the host side, figuring out what files need to be queried for errors and when. Another thing to note, is that there are trade offs, you can not be both complete and fast, so you can not aim at getting all errors immediately after an edit, again that does not scale with bigger projects.
Nov 26, 2013 at 12:32 AM
Thanks for the info - sounds like there is no magic bullet for this.
Nov 26, 2013 at 10:45 AM
I had been wondering about this, too, as in: which files do need to be re-processed when there is a change.

Part of the problem is that external modules are not yet complete or popular. And with reference-only projects, there is no precise dependency tree - every file that was referenced before can cause breakage later on.

As long as the current edit did not change imports or references, one can report errors on the current file first, looking at dependencies (or files later in the reference order) later. Otherwise, re-resolving the imports and references becomes necessary. Which makes me wonder why the resolver and the language services aren't integrated more closely (currently, they don't know of each other - there is no guarantee that the host integrates them correctly)?
Nov 26, 2013 at 1:33 PM
Few things I would look into:

1) Make sure to keep the whole AST in memory and use language service API to update only changed files. TS has a effective update mechanism for partial updates, but of course means you have to keep the state of the language service in between.

2) Offload the work to a separate thread to do the work. I work on a web based IDE and use a (state full) web worker to do all the hard stuff. To be honest it is quicker than I expected.

3) For external referenced files you could find out the dependency:
            var i = TypeScript.ScriptSnapshot.fromString(script.content);
            var refs = TypeScript.getReferencedFiles(script.fileName,i);

And based on a dependency tree you could first get the Diagnostics for the most likely impacted files.
Nov 26, 2013 at 5:07 PM
There tends to be no proper dependency tree if references are used, only a position in a list of references (you can use references as explicit dependencies; you can use references as includes, with implicit dependencies; you can use references to manage a project-wide targets list - every source file references a single project-wide file that references all project files; or any mix of these).

Any files that come before the changed file in that reference list might be obsoleted by modified references in that file (but it is hard to know which ones, without resolving the whole project again). Any new references in the changed file might have to be added to the reference list. Any file that come after the changed file in the reference list might be affected wrt diagnostics. It is even possible to cause errors in not directly related files (files that neither reference the modified file, nor are referenced by it), just by affecting the order of references in that list.

One should probably do a best-effort thing and ignore the corner cases - my thinking just tends to get stuck on these;-)
Nov 27, 2013 at 11:57 AM
Edited Nov 27, 2013 at 11:57 AM
Another case (that I often use) is to not use reference at all ! I personally use something like :
tsc MyDeclFile1.d.ts MyDeclFile2.d.ts MyDeclFile3.d.ts .... myMainFile.ts 
for compiling whithout any reference ^^"
Dec 8, 2013 at 2:10 PM
Edited Dec 8, 2013 at 2:12 PM
It is perhaps something stupid but I'm trying actually an approach that could allows to retrieve true 'file dependencies' :
class GetDependenciesWalker extends TypeScript.PositionTrackingWalker    {
    
    static getDependencies(fileName: string,  languageService: Services.ILanguageService) {
                
        var dependencies = new collections.StringSet(),
            syntaxTree = languageService.getSyntaxTree(fileName);
        
        syntaxTree.sourceUnit().accept(new GetDependenciesWalker(fileName, languageService, dependencies));
        return dependencies.keys;
    }
    
    constructor(
        private fileName: string, 
        private languageService: Services.ILanguageService, 
        private dependencies: collections.StringSet
    ) {
        super();
    }
    
    public visitToken(token: TypeScript.ISyntaxToken): void {
        if (token.kind() === TypeScript.SyntaxKind.IdentifierName) {
            var definitions = this.languageService.getDefinitionAtPosition(this.fileName, this.position());
            if (definitions) {
                definitions.forEach(definition => {
                    if (definition.fileName !== this.fileName) {
                        this.dependencies.add(definition.fileName);
                    }
                });
            }
        }
        super.visitToken(token);
    }
}
I'm afraid however about the performance, if anybody has a better idea, let me know