feat: use compiler tracers for project indexing#764
Open
doorgan wants to merge 4 commits into
Open
Conversation
doorgan
force-pushed
the
doorgan/compiler-tracers-2
branch
4 times, most recently
from
July 15, 2026 03:25
eb8fb1d to
d8aa44f
Compare
doorgan
marked this pull request as ready for review
July 15, 2026 17:39
The main idea of this PR is that we can use Compiler Tracers for all compiled files like NextLS does. This allows us to index definitions and references *as the project compiles* rather than as a separate step. As a consequence, the indexer no longer goes over .ex files, as those are compilable, and indexes all .beam files instead if a compiler tracer missed them. Compiler tracers provide all the information we need: aliases, alias references, remote/imported functions/macros, local functions/macros, struct expansion, and definitions(via the module's compiled bytecode), which in principle should allow us to skip indexing from source code, or in the worst case scenario reduce the source analysis to range refinement, which is faster than full AST analysis. For non-compiled files like .exs scripts and tests, we still run the source code based indexer so we don't lose support for those. At a high level this works as follows: - File is saved, compilation is scheduled - The compiler emits tracer events, we pick them up and update both the index and the indexer manifest - The indexer runs, skips the .beams from files already analyzed via compiler tracer events, indexes the rest of the stale files if any If the index is missing but the project was already compiled, we will index from disk files instead: from .beam files for definitions since those are generally way more complete, and from the .ex sources for references. --- This introduces two different compiler tracers(just like NextLS). This is because mix compilation happens in two main stages: 1. Deps compilation via `deps.loadpaths` 2. Actual compilation of the user code This happens both in Expert and in regular `mix compile` runs, as `deps.loadpaths` is called internally by `mix compile`. We need to both run a compiler tracer for dependencies, AND to discriminate between dependency code and user code to decide wether or not to index references. Having a DependencyTracer that is configured only during `deps.loadpaths` and a ProjectTracer that is configured afterwards allows us to do both things. There is another quirk with `deps.loadpaths` as of elixir 1.20 where the compiler options will not get propagated to the compiler workers if `MIX_OS_DEPS_COMPILE_PARTITION_COUNT` is `>1`, fixed in elixir's main branch with commit `https://github.com/elixir-lang/elixir/commit/6f18bf70c07b118dc434b3fd478b420071bc487f` and not yet released in a stable version. To workaround this issue, that env var is forced to `1` for the duration of the compilation. --- Later on for 0.4 we will want to investigate using the compiler locks to share the expert build with the user builds and save repeated work. That would be partially incompatible with compiler tracers because a user triggered build will not send the compiler tracer events to expert. In that case we want to also have the ability to index what's missing after compilation, and this PR should set the foundations for that. I'd like to know how this perform in real codebases too, as I see a noticeable slowdown in cold compilation(mostly the `deps.loadpaths` step), but I don't know how impactful this is on large codebases.
doorgan
force-pushed
the
doorgan/compiler-tracers-2
branch
from
July 15, 2026 17:45
d8aa44f to
0c7992c
Compare
Contributor
Author
|
I think the issues are fixed now, in particular find references which was reported as not working: For a module For a public function For private function For module attributes For variables For erlang modules For guards For macro generated functions For all the above, go to definition also works. One reported issue was that references for modules and functions were not working. This new PR does not have commits that specifically fix those, so I'm not really sure what happened in that case and I can't reproduce the problem as you see above. I'd appreciate more eyes testing this! |
scohen
reviewed
Jul 15, 2026
Comment on lines
+268
to
+272
| defp timed(fun) when is_function(fun, 0) do | ||
| start = System.monotonic_time(:millisecond) | ||
| result = fun.() | ||
| {System.monotonic_time(:millisecond) - start, result} | ||
| end |
Contributor
There was a problem hiding this comment.
isn't this just :timer.tc but with millis? Why not just use timed_log ?
Contributor
|
Running this branch, I get "mix deps.loadpaths" as progress during deps compilation, not the actual files being compiled. |
Contributor
|
again, there's a dramatic slowdown during compilation of our largest file. It's still taking ages. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a re-publishing of #705, which was reverted due to issues we found after merging. You can read that PR for the original description of how this works.
Adds the following fixes on top of the previous attempt:
GenServer.castinstead ofGenServer.callin theTraceBuffer, this should make compilation faster by not serializing the work done with the tracer events. This was the original intent but I missed it in the original implementation, as the timing didn't noticeable change when testing it against expert itself.