Develop: LanguageService.cleanupSemanticCache()

Topics: General
Oct 30, 2013 at 5:47 PM
I noticed a new function has been added to the develop branch: LanguageService.cleanupSemanticCache(). Should this method be called before getting diagnostics across all files or is it for some other scenario?
Developer
Oct 30, 2013 at 11:36 PM
Hi Derek,

The purpose of this method is to let the host tell the compiler that it has performed the large semantic operations it intends to perform and the compiler can throw away semantic data. You should call it after you're done getting all diagnostics for all files and if you expect to not need semantic information afterwards.

So, in practice, you'd use this for projects with no files that are currently opened by the user. Since the user isn't editing the files, storing around all that semantic information is wasteful once we've computed diagnostics.

It's optional to use, but is handy if you're dealing with lots of projects, or very large projects.

I hope that helps clarify things. Please let me know if you want more info.

Thanks!
Oct 31, 2013 at 1:36 AM
Interesting, so it sounds like it allows the host to manage the memory usage a bit more granularly. I'm having trouble coming up with a scenario in which say an IDE would want to call this though. For example, in the current Eclipse plug-in code, when a file is saved, the plug-in asks for diagnostics across all files (in case a change in that one file impacted others). Wouldn't it make sense to always retain that cached semantic information for subsequent file save operations to avoid recomputing the information if possible?
Developer
Oct 31, 2013 at 4:18 AM
"Wouldn't it make sense to always retain that cached semantic information for subsequent file save operations to avoid recomputing the information if possible?"

The language service and compiler already preserve whatever information is valid to store across edits. Semantic information does not apply. The semantic information from before is always recomputed as nearly any edit can/will affect it.

"I'm having trouble coming up with a scenario in which say an IDE would want to call this though."

As mentioned above, the scenario is:

You have a solution with many projects. Most of the projects are not being edited by the user. You've collected diagnostics for them, but now don't expect to need anything else until the user starts editing it. Instead of having a lot of memory allocated and held around unnecessarily, you can allow the engine to reclaim all that memory.

"For example, in the current Eclipse plug-in code, when a file is saved, the plug-in asks for diagnostics across all files"

This appears to be the difference between how Eclipse consumes the compiler and how VS does. You have a single compiler instance apparently used across all files. In VS this is not how we do things. We have a compiler instance per VS project. For your use case, where you merge all files together, this method would not be helpful. For a use case where there are lots of independent islands of files, this can be very helpful.

Hope this helps!
Oct 31, 2013 at 6:37 AM
Oh interesting - thanks for the explanation - I was wondering how VS supports multiple projects.

The Eclipse plug-in also currently has a separate language service instance per project but it doesn't support references across projects yet. That's going to be my next project now that we've got our code compiling with the develop branch compiler. Separate compilers for each project makes sense since each project will have different files (and dependencies loaded).

Any advice for getting AMD module lookups to work across projects? (I'm guessing some tricks with the language service host's path resolution logic)
Developer
Oct 31, 2013 at 8:17 AM
For cross project cases, i would generate the .d.ts for the project and add that file to the projects that you want to be referencing it. In that way a .d.ts acts as the exported surface area that the other project should know about.
Oct 31, 2013 at 12:33 PM
This is very useful information. Why not have a blog dedicated to language service development? Or a mailing list, a wiki with FAQ, etc?

Anything to make life easier for language service clients (of which there are half a dozen now), by improving communication;-) The whole point of the language services is that TS tools can share code/ideas.

I develop/maintain typescript-tools (basis for Vim, Emacs, and Sublime plugins), and I do not always have the option of following TS commits as they happen. Reconstructing intentions and necessary client code change from commits is no fun, even less so when a few weeks have gone by and the "last known good" isn't that good after all.
Oct 31, 2013 at 5:43 PM
@cyrusn We are doing the d.ts thing right now but it doesn't scale well for AMD projects because it requires generating a d.ts for each module (and then keeping them up-to-date in all the projects containing them). That's why we'd like to build against the original sources across projects.
Developer
Oct 31, 2013 at 9:28 PM
Sounds like a great idea Claus. The main problem right now is simply one of resources. I'm going to forward this along to our PM to see if we can track doing something like this and getting better communication with the community.

Thanks!
Developer
Oct 31, 2013 at 9:31 PM
@derek. Hrmmm. Could you clarify why it doesn't scale? I'm not quite getting it.

Note: it will be the case (if you have project references) that a chance in one project may end up causing you to need to get diagnostics for all dependent projects (if hte .d.ts surface across referenced projects). I'm not sure if there's any way around that. That's simply the nature of the language and the transitive impact that changes cause.

Thanks!
Oct 31, 2013 at 11:38 PM
Right now we have one core project that contains basic functionality like communication with the server. We plan to have several other projects that make use of the functionality in that core project. If we generate a d.ts file for each module used by those other projects we could easily be up to hundreds of files we'd have to keep in sync. If we could find a way to reference the source files directly across projects, then we wouldn't need to keep anything in sync (it would be more like Java or C# projects).
Developer
Oct 31, 2013 at 11:50 PM
@derek

i guess i'm not seeing why this is particularly difficult. but i don't know the particulars of your domain/architecture. :-)

A project changes. You generate the d.ts file for it. You inform dependent projects. You generate their d.ts. files, and so on down the dependency graph.

Note: this is the world today already in VS for things like VB and C# and C++ projects. 'metadata dlls' are the lingua franca used for one project to tell another "here's my public surface area you can use". VS and the build system know the dependency graph and pass along the information between projects as changes happen.

We could try to provide something like this ourselves. But it won't prevent any "scaling" issues. It will just push those scaling issues to a different part of the stack. The reason i like it above the compiler is it allows compilation of one project to happen in isolation of all others instead of having one compiler need to reach into data held by another.
Developer
Oct 31, 2013 at 11:56 PM
Note: i'm not dismissing the idea. It's definitely something to noodle on. I'm just not seeing slam dunk benefits to it.
Oct 31, 2013 at 11:59 PM
Hmm, interesting. Are you talking about generating d.ts files automatically via the plug-in and injecting them into the downstream projects or creating them manually? Right now we are manually building d.ts files for our core project and copying them into the dependent projects. We have to repeat this process each time the core changes its public APIs. Also, there can be several d.ts files which need to be exported from a single project since we are using AMD modules (because they are referenced by their file paths vs. internal module names). Currently I think we export 3 d.ts files from our core project since there are 3 modules used by other projects. If we have say 10 dependent projects, that would be 30 files to keep up-to-date.
Developer
Nov 1, 2013 at 12:01 AM
I was definitely thinking about them being 'auto generated'. I agree the 'keep them up to date yourself' approach wouldn't scale :)
Nov 1, 2013 at 12:05 AM
Hmm, that's a really interesting idea. I honestly never thought of something like that. Would you generate them to disk so that they actually exist as files in the downstream projects or just generate them into the language service host somehow? Either way it seems like it would be necessary to somehow mark certain modules as being exported so that the plug-in would know to build them.
Developer
Nov 1, 2013 at 12:08 AM
In general, i would nt actually pollute the disk where real project files were stored. I would either do it in memory, or use a temp location if i really needed the disk.

The LS API has no knowledge of the disk. It just talks to the host to get file contents. So it leaves how/where the files are stored completely up to the host.
Nov 1, 2013 at 12:15 AM
Cool, that's a neat idea. I'll have to think about this a bit more - one problem off the top of my head is that this would break the ability to follow the definition of something to its source to inspect the implementation (something we do during Java development all the time). But the approach of injecting things into the LS host in general seems like a good path to go down.
Developer
Nov 1, 2013 at 12:32 AM
True. One thing that would be nice to provide (and inspired by the work in the Roslyn project) would be a way to go from a definition in d.ts to the original symbol that generated it. That way you could say "oh... goto def took me to this d.ts. method let me see if that was backed by source and take them there".

No plans on that yet. but it would certainly be nice to have in the future :)
Nov 1, 2013 at 12:33 AM
@cyrusn if you want to hire outside help, let me know;-)

related feature suggestions that would be useful:
Nov 1, 2013 at 11:12 AM
As for jump-to-def: isn't that already problematic for interfaces, even without .d.ts files? When I apply my LS client to itself, and ask the LS for the definition of language service methods, I get directed to the LS interface, with no obvious way to go to the pullLS implementation.

PS. I would usually log all these things on the issue tracker, but workitems all too often get closed if they are not in scope for TSv1.0.
Developer
Nov 1, 2013 at 11:14 AM
Yup. It's a similar issue to C#.

If you're calling through an interface method, then the def is the interface method. Going from def to implementation would definitely be a nice-to-have though.