Make Compiler a stdlib #56409
Open
Make Compiler a stdlib #56409
+390
−262
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is a further extension to #56128 to make the compiler into a proper stdlib, useable outside of `Base` as `using Compiler` in the same way that `JuliaSyntax` works already. There's a few remaining questions around how loading works, but mechanically, this PR is complete. For those remaining questions, I'll probably put up a separate PR that would migrate JuliaSyntax to it and then simply adopt that here once we've figured out the correct semantics.
This tries to align the semantics of `using Compiler` with that of all other (upgradable, not-in-the-sysimage) stdlibs by semantically treating the Compiler as a non-loaded stdlib that just happens to already have its `.ji` file included in the sysimage. Then, when you ask it to create a `.ji` file, it simply writes out a stub file that redirects everything to the sysimage. Example: Here we load an unmodified compiler image. Everything is done in 1s and the `Compiler` you get back at toplevel is the same Compiler that Base uses. ``` julia> @time using Compiler ┌ Debug: Rejecting cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.ji for Base.PkgId(Base.UUID("807dbc54-b67e-4c79-8afb-eafe4df6f2e1"), "Compiler") since pkgimage /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.so was not found └ @ Base loading.jl:3945 ┌ Debug: Rejecting cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_ajVhJ.ji for Base.PkgId(Base.UUID("807dbc54-b67e-4c79-8afb-eafe4df6f2e1"), "Compiler") since the flags are mismatched │ requested flags: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2 [163] │ cache file: use_pkgimages = true, debug_level = 2, check_bounds = 0, inline = true, opt_level = 2 [165] └ @ Base loading.jl:3918 ┌ Debug: Rejecting cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.ji for Base.PkgId(Base.UUID("807dbc54-b67e-4c79-8afb-eafe4df6f2e1"), "Compiler") since pkgimage /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.so was not found └ @ Base loading.jl:3945 ┌ Debug: Rejecting cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_ajVhJ.ji for Base.PkgId(Base.UUID("807dbc54-b67e-4c79-8afb-eafe4df6f2e1"), "Compiler") since the flags are mismatched │ requested flags: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2 [163] │ cache file: use_pkgimages = true, debug_level = 2, check_bounds = 0, inline = true, opt_level = 2 [165] └ @ Base loading.jl:3918 ┌ Debug: Rejecting cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.ji for Base.PkgId(Base.UUID("807dbc54-b67e-4c79-8afb-eafe4df6f2e1"), "Compiler") since pkgimage /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.so was not found └ @ Base loading.jl:3945 ┌ Debug: Rejecting cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_ajVhJ.ji for Base.PkgId(Base.UUID("807dbc54-b67e-4c79-8afb-eafe4df6f2e1"), "Compiler") since the flags are mismatched │ requested flags: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2 [163] │ cache file: use_pkgimages = true, debug_level = 2, check_bounds = 0, inline = true, opt_level = 2 [165] └ @ Base loading.jl:3918 [ Info: Precompiling Compiler [807dbc54-b67e-4c79-8afb-eafe4df6f2e1] (cache misses: mismatched flags (2), missing ocachefile (2)) ┌ Debug: Generating object cache file for Compiler [807dbc54-b67e-4c79-8afb-eafe4df6f2e1] └ @ Base loading.jl:3021 ┌ Debug: Loading object cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.so for Compiler [807dbc54-b67e-4c79-8afb-eafe4df6f2e1] └ @ Base loading.jl:1277 1.000245 seconds (507.60 k allocations: 29.245 MiB, 1.68% gc time, 65.46% compilation time) shell> ls -la /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.so -rwxrwxr-x 1 keno keno 7112 Nov 1 07:33 /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.so julia> Compiler === Base.Compiler true ``` Next, we dev the Compiler from another Julia version. Now it just treats is as a regular package load and generates a 5MB .ji that contains the usual pkgimage. ``` (@v1.12) pkg> dev /home/keno/julia2/stdlib/Compiler Resolving package versions... Updating `~/.julia/environments/v1.12/Project.toml` [807dbc54] + Compiler v0.0.1 `~/julia2/stdlib/Compiler` Updating `~/.julia/environments/v1.12/Manifest.toml` [807dbc54] + Compiler v0.0.1 `~/julia2/stdlib/Compiler` julia> using Compiler ┌ Debug: Rejecting stale cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.ji because file size of /home/keno/julia2/stdlib/Compiler/src/Compiler.jl has changed (file size 5958, before 5949) └ @ Base loading.jl:3875 ┌ Debug: Rejecting cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_ajVhJ.ji for Base.PkgId(Base.UUID("807dbc54-b67e-4c79-8afb-eafe4df6f2e1"), "Compiler") since the flags are mismatched │ requested flags: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2 [163] │ cache file: use_pkgimages = true, debug_level = 2, check_bounds = 0, inline = true, opt_level = 2 [165] └ @ Base loading.jl:3918 ┌ Debug: Rejecting stale cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.ji because file size of /home/keno/julia2/stdlib/Compiler/src/Compiler.jl has changed (file size 5958, before 5949) └ @ Base loading.jl:3875 ┌ Debug: Rejecting cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_ajVhJ.ji for Base.PkgId(nothing, "") since the flags are mismatched │ requested flags: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2 [163] │ cache file: use_pkgimages = true, debug_level = 2, check_bounds = 0, inline = true, opt_level = 2 [165] └ @ Base loading.jl:3918 Precompiling Compiler finished. 1 dependency successfully precompiled in 3 seconds ┌ Debug: Loading object cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.so for Compiler [807dbc54-b67e-4c79-8afb-eafe4df6f2e1] └ @ Base loading.jl:1277 shell> ls -la /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.so -rwxrwxr-x 1 keno keno 5706792 Nov 1 07:40 /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.so julia> Compiler === Base.Compiler false ```
|
Ok, I think I've come up with something clever for the loading semantics. This tries to align the semantics of Here we load an unmodified compiler image. Everything is done in 1s and Next, we dev the Compiler from another Julia version. Now it just treats |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a further extension to #56128 to make the compiler into a proper stdlib, useable outside of
Baseasusing Compilerin the same way thatJuliaSyntaxworks already. There's a few remaining questions around how loading works, but mechanically, this PR is complete. For those remaining questions, I'll probably put up a separate PR that would migrate JuliaSyntax to it and then simply adopt that here once we've figured out the correct semantics.