LanguageService an incrementalCompilation problems

Topics: General
Apr 16, 2014 at 11:17 PM
I'm developing an extension for Brackets (editor) that add TypeScript support.
I'm experiencing painful experience with feeding the compiler with edit.
Sometimes the languageService will send errors like if the sourceTree is desynchronized, findToken operation on retrieved syntax tree will throw errors. However when I check the string content stored in my languageService host it is in sync with my editor.
Here is my ScriptSnapshot implementation, it's basicly a modified version from the one in test harness, if anybody could give me at least a hints on what I do wrong it would helps :


/**
 * Manage a script in the language service host
 */
class ScriptInfo {
    ...
    editRanges: TypeScript.TextChangeRange[] = [];
    ...
    updateContent(newContent: string): void {
        this.setContent(newContent);
        this.editRanges = []
        this.version++;
    }
    ....
    editContent(minChar: number, limChar: number, newText: string): void {
        // Apply edits
        var prefix = this.content.substring(0, minChar);
        var middle = newText;
        var suffix = this.content.substring(limChar);
        this.setContent(prefix + middle + suffix);

        // Store edit range + new length of script
        this.editRanges.push(
            new TypeScript.TextChangeRange(
                TypeScript.TextSpan.fromBounds(minChar, limChar),
                newText.length
             )
        );

        // Update version #
        this.version++;
    }

    getTextChangeRangeBetweenVersions(startVersion: number, endVersion: number)  {
        if (startVersion === endVersion) {
            // No edits!
            return TypeScript.TextChangeRange.unchanged;
        }

        var initialEditRangeIndex = 
                this.editRanges.length - (this.version - startVersion);
        var lastEditRangeIndex = 
                this.editRanges.length - (this.version - endVersion);

        var entries = this.editRanges.slice(
            initialEditRangeIndex,
            lastEditRangeIndex
         );
        if (entries.length === 0) {
            return null;
        }
        return TypeScript.TextChangeRange
            .collapseChangesAcrossMultipleVersions(entries);
    }
}
class ScriptSnapshot implements TypeScript.IScriptSnapshot {
    private scriptInfo: ScriptInfo;
    private lineMap: TypeScript.LineMap = null;
    private textSnapshot: string;
    private version: number;

    constructor(scriptInfo: ScriptInfo) {
        this.scriptInfo = scriptInfo;
        this.textSnapshot = scriptInfo.content;
        this.version = scriptInfo.version;
    }

    getText(start: number, end: number): string {
        return this.textSnapshot.substring(start, end);
    }

    getLength(): number {
        return this.textSnapshot.length;
    }

    getLineStartPositions(): number[] {
        if (this.lineMap === null) {
            this.lineMap = TypeScript.LineMap1.fromString(this.textSnapshot);
        }
        return this.lineMap.lineStarts();
    }

    getTextChangeRangeSinceVersion(scriptVersion: number): TypeScript.TextChangeRange {
        return this.scriptInfo.getTextChangeRangeBetweenVersions(scriptVersion, this.version);
    }
}
 
Developer
Apr 17, 2014 at 5:16 PM
Hey there,

I'm not seeing anything jumping out a obviously wrong. One thing that might be an issue is 'updateContent'. I've noticed when that is called that you clear our the array of edits. Now, imagine the following sequence of events:

a) 3 edits.
b) 1 update.
c) 3 edits.

With your code as is, the array will be cleared out, and then will have 3 entries in it. Now, let's say the LS asks for the changes between version 2 and version 6. The slice in getTextRangeBetweenVersions will get the wrong text change and will end up borking things pretty badly.

What you can do instead is copy the array of changes over whenever you make a new ScriptSnapshot. That array's last element will correspond to the change that created that version of the script snapshot. Then, when you ask for the changes between versions, you can verify that the array you have has enough changes in it to satisfy that request. If it does, you can return the right value. Otherwise, you can return null to indicate that you don't know the change, and that the LS should reparse the entire file.

I hope that that helps!
      -- Cyrus 
Apr 17, 2014 at 6:19 PM
Thanks a lot for giving a look, I will try that.
Developer
Apr 17, 2014 at 6:51 PM
Great!

Also, i wanted to mention something explicitly (in case it wasn't clear).

If you ever end up doing one of your 'updateContent' operations, you are essentially saying "i can no longer accurately determine the text change range across that version number". i.e. if you do the updateContent on version 5, then any request that crosses version 5 will need to return 'null' to say "i don't know what changed". The TypeSCript LS will then be very conservative and will simply reparse the file entirely.

We recognize this is a complex part of the existing system, and we're working on a mechanism to make this simpler in the near future.

First off, instead of passing you a version number to determine changes for, we will instead pass you the two ScriptSnapshot instances themselves. If your editor can then efficiently determine the changes between these two versions, you can use that mechanism (for example in VisualStudio, we can use the ITextVersion data the editor stores to know precisely what changed between any two versions).

Second, we will possibly make this operation optional for the host (i.e. you). And, if you don't provide it, we'll do a diff of the snapshots to figure out what we need to reparse. This won't be as efficient as you just answering the question, but it may be good enough for your use cases. And, if it turns out to be too slow, you can always swap in a more efficient implementation later. In general, determining the set of changes between two documents of size A and B is O(AB). However, we can make a few adjustments to make that much closer to linear while still providing great results on the types of edits that normally happen in a document.
Apr 17, 2014 at 7:12 PM
Edited Apr 17, 2014 at 7:13 PM
In fact your solution seems to work perfectly apparently this was the problem. Thanks a lot for your help, this bug gave me a big headache. I now clone editRanges on the snapshot and return null if I have less ranges than asked by the compiler.