Where do IDEs get completion candidates from?

Question

Where? Can someone give me example with source retrieval methods in the IDEs of these popular languages:

• C/C++ (Visual Studio, Eclipse): Do they use static analysis - built into the IDEs - based on project root (with configuration files, i.e. for include paths) to retrieve candidates? Or do they use compiler backend to get the source, i.e. clang? How does IDE like Visual Studio get completion candidates system wide like Window API and do it so quickly?
• Java (Netbean, Eclipse): Again, do they use any of java binaries (java, javac...) in the JDK or a built-in static parser?
• Javascript: the same as above

Answer 1

Major IDEs such as Visual Studio, Eclipse and IntelliJ Idea use indexing approach: parsing all of the project files and storing possible completion symbols(i.e. method names). So, when the user presses Ctrl+Space to invoke completion list, the editor looks up through the stored symbols to filter the most appropriate candidates.

These kind of source code parsers named "incremental parsers" and they are distinguish from the ordinary parsers by two features:

1. Incremental parser works continuously. It must always be in touch with the source code's syntax. So the index of completion symbol candidates is always up to date.
2. Small and frequent changes that the end user makes to the source code should be indexed without any significant time delays.

Incremental parsers reach these goals by utilizing some form of caching. This visual demo illustrates how they work in a nutshell: Incremental JSON parser. I recommend you to try it.

Also you may be interested in reading these sources:

1. An article about source code indexing architecture in IntelliJ Idea: Language plugins in IntelliJ Idea.
2. Eclipse built-in Java incremental parser: Eclipse JDT.

Unfortunately, there is a real lack of tools to build your own incremental parser. Core parsers of the most IDEs are hardcoded, and they don't share any common approach. The demo I mentioned above is one of the rare exceptions that uses distinct incremental parser library - Papa Carlo for building parsers in Java, Scala and JavaScript. Another example is Parsley - incremental parsing library for Clojure language.