Parallel builds support #392
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Cool thanks for looking into this! |
indygreg
force-pushed
the
gps/parallel-builds
branch
from
74a9dc0 to
76cdbd9
Compare
|
The 3.13 CI jobs aren't yet fully optimized and already this PR drops Linux builder total time from ~64h to ~43h, or ~67% of baseline. Not bad. |
I'm about to enable support for parallel CPython builds. This function was currently not safe if called in parallel. This was causing setuptools / pip downloads to race writing to the same file and failing builds.
This is less ergonomic (oh Make). But it will make it easier to build multiple Python versions at the same time in a future commit.
Previously we generated the same Makefile for every Python version. We can simplify things by only writing 1 Makefile per target configuration.
This refactor allows targets for each major Python version to be defined simultaneously. This opens the door to a single `make` invocation building multiple versions in parallel.
`build-main.py` now passes in space delimited Python versions to build, which the Makefile can happily translate to appropriate make targets for us. We added support for `--python all` to allow building all supported Python versions.
Our build system supports this now. So update CI to make use of the feature. On Linux, 3.13 freethreaded builds are still their own CI job. We may need to tweak `build-main.py` to allow intelligently selecting freethreaded compatible versions. We could potentially also allow building freethreaded in parallel with non-freethreaded. This can be done as a followup.
indygreg
force-pushed
the
gps/parallel-builds
branch
from
76cdbd9 to
d17e9dc
Compare
|
The macOS builds should be fixed. Issue turned out to be a compatibility issue changed in GNU Make 3.82 (macOS ships 3.81). Specifically commit 81f3e4babd128f6740d05b371122762924522fb6 (because I bisected it).
|
|
Ok. CI came back clean. macOS CI total time drops from 10:27 to 8:49. 529/627=84% of baseline. Not the percentage savings as Linux. And 3.13 is still on the table for optimization. But still a nice win. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is an attempt to support building multiple Python versions at the same time. The impetus for this is reducing the amount of work that CI does by amortizing the costs for building common dependencies.
The first 3 commits should be relatively safe to merge. 4 and 5 need a bit of testing.
Open issues: