You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As long as ninja is the only build execution tool, the current ninja -jN implementation works fine.
But when you try to convert parts of an existing recursive GNU make based SW build system to ninja, then you have the following situation:
top-level GNU Make (with -jX, acts as job server)
M instances of GNU make (with -j, act as job server clients)
N instances of ninja (don't know anything about job server)
Simply calling `ninja -jY' isn't enough, because then the ninja instances will try to run Y*N jobs, plus the X jobs from the GNU make instances, causing the build host to overload. Relying on -lZ to fix this issue is sub-optimal, because load average is sometimes too slow to reflect the actual situation on the build host.
It would be nice if GNU make jobserver client support could be added to Ninja. Then the N ninja instances would cooperate with the M GNU make instances and on the build host only X jobs would be executed at one time.
Reacted by Fabio Porcedda, Aleksandr Mezin, Joel Rosdahl, Jürg Billeter, Loïc Yhuel, comicfans, Michael Corcoran, corse-master, Kevin Gaudet, Segev Finer and 81 more
I have tested this implementation over the last few weeks in two different recursive GNU make based build systems that originally had M+1 GNU make instances:
use case A: top-level GNU make, 1 ninja instance, M-1 GNU make instances
use case B: top-level GNU make, N ninja instances, M-N GNU make instances
FYI: google/kati was used to convert existing single makefile GNU make parts to Ninja build file.
We've discussed this on the mailing list a few times (e.g. here https://groups.google.com/forum/#!searchin/ninja-build/jobserver/ninja-build/PUlsr7-jpI0/Ga19TOg1c14J). Ninja works best if it knows about the whole build. Now that kati exists, one can convert those to ninja files and munge them up to have a single build manifest (that's Android's transition strategy from Make to Ninja -- they use kati to get everything converted to Ninja files, and then they're incrementally converting directories to use something-not-make -- and then kati produces parts of their Ninja files and the new thing produces parts of the ninja files.)
Is your use case that you have recursive makefiles?
Reacted by Ross Smyth
Reacted by pal666, Victor Sergienko, StefanBruens, cheshirekow, Mauro Toffanin, Michał Dominiak, Johan Boulé, Kunal Tyagi, Jason Rhinelander, Ivan Shapovalov and 3 more
I could have guessed that this has been discussed before, because I'm surely not the first person facing such a situation.
Here are my reasons for requesting this:
recursion: kati currently can't translate recursive GNU make based build systems, like Linux kernel kbuild. IMHO a major effort and unfortunately I can't wait for kati to provide this, hence the such sub-component builds will have to stay with GNU make for the time being.
missing features: kati currently can't translate fully modularized GNU make based build systems, i.e. where each component is built in isolation and in a separate build directory, so that all ninja.build files could be merged into a single one. While IMHO not such a major issue as (1) it is much simpler to replace the lowest-level $(MAKE) recipe with a kati/ninja recipe. Parsing + merging might also introduce unnecessary build delay (needs to be seen what would happen in real life)
technical barriers: e.g. sub-component builds that run behind a "chroot firewall". Even if everything moves to Ninja, you would still need 1 (main) + N (one for each chroot) ninja instances that need to cooperate. Ninja doesn't offer anything like that.
too simple workarounds: AOSP makeparallel + kati/ninja runs all $(MAKE) instances hard-coded with "make -j4" with no cooperation between any of the GNU make instances. That is only acceptable if you have no or only a few or small $(MAKE) invocations from the ninja.build file.
organizational barriers: even if it might be possible to use kati/ninja to convert an existing GNU make base sub-part of the system, you might not be allowed to do so. Such sub-component builds need to stay with GNU make.
You ask: why not split the build up and run them as separate builds? Goto (5)...
IMHO my patch provides a good solution, considering
how small the required changes to ninja are,
that the default behaviour is completely unchanged, and
that this will make the life easier for many other ninja users which face the same issues
Reacted by noseglasses, Wilfried Goesgens, Curtis J Bezault, StefanBruens, cheshirekow, Axel Huebl, Cosmin Stejerean, Jia Yuan Lo, Michał Dominiak, Serhii Pavlovskyi and 18 more
Another possible reason for having jobserver in ninja seems to be LTO support in gcc. -flto=jobserver tells gcc to use GNU make's job server mode to determine the number of parallel jobs. The alternative is to spawn a fixed number of jobs with e.g., -flto=16.
Reacted by Ellen Emilia Anna Zscheile, Vladimír Vondruš, StefanBruens, Avi Kivity, cheshirekow, Axel Huebl, Nicolás Alvarez, Jake Merdich, Yousong Zhou, Serhii Pavlovskyi and 17 more
Any suggestions for such a multi-build? Any recursive Ninja invocations (either from Meson or CMake) would fit the build I think. Or maybe something that uses Make at the top-level, but then builds multiple things with Ninja.
You'll want to use the selftest subdirectory and to enable some selection of projects (the compression libraries are probably a good candidate set). To get mismatched generators (i.e., different generators for the "superbuild" compared to the inner builds, specify CMAKE_GENERATOR "inner name" in the _superbuild_ExternalProject_add call in cmake/SuperbuildMacros.cmake (line 1550 or so). There's also some logic that may need tweaked to understand that ninja can also provide a jobserver around line 30 in cmake/SuperbuildExternalProject.cmake.
so this "degraded performance"... is it just that I came up with a scaling method for capacity instead of "infinite"? then the performance would be the same if I put it back to infinite perhaps?
That's what I tried initially with your PR, but the resulting binary didn't provide any improvements to my build, except very reduced volatility. Here are the corresponding raw numbers I had at the time for a tiny Fuchsia build, in case you are interested:
But note that with your latest commit, things are far worse, unfortunately. You didn't even explain why you decided to change the capacity logic, nor provide any meaningful numbers for its usefulness. And again, you didn't provide any tests verifying the basic correctness of your code.
I know you have worked considerably on this, and addressed a lot of review comments, but you didn't address all the issues/ The sad truth is that this was still not good enough, even performance wise, at least for my use case. Hence the motivation to start something else.
maybe I should have mentioned my system has only 4 cores
the jobserver protocol is supposed to be simple but what I'm seeing instead is as verbose and unreadable as you can possibly make it, pulling out all the tricks you have ever learned, for what benefit honestly?
don't get me wrong, I'm sure it works great, it just looks vastly over-engineered at first glance
That's just your opinion.
At this point, I am more interested in finding a common ground that we can all use to verify that whatever lands into the final Ninja tree works correctly for everyone.
Because PR#2506 works for me doesn't mean it would work for you or other people, hence why it is so important to get good common regression tests. here. And fix whatever needs to be fixed otherwise.
That's what I tried initially with your PR, but the resulting binary didn't provide any improvements to my build, except very reduced volatility
Ah ok... so there is much more going on here...
You didn't even explain why you decided to change the capacity logic, nor provide any meaningful numbers for its usefulness
I was following the initial complaint or concern about the value being "infinite", so I tried to determine a working finite value, or at least I thought it was working, that coincides with the whole reason that the "capacity" value was set up in the first place in 8a2575e, in order to gently reach maximum parallelism without an instantaneous load which can shock the system. What you're calling "volatility" in this exact case may be on purpose... or am I misunderstanding something?
The value I put is supposed to represent a capacity of 1 + 2^n with n incrementing for every loop of the builder per finished edge until the next step is the maximum. Because my ability to test high parallelism is limited, it could be that it's actually 1 + 2*n instead...
So do you know which optimization is improving the performance and where? I would assume it has something to do with how the "acquire" and "release" functions handle the data. I would like to learn something and be corrected and informed instead of just bypassed. Perhaps extreme optimizations are necessary in one or a few spots, but it would be nice if we can determine where it actually matters so other spots in the code can return to simplicity. I understand that in the world of optimization, speed requires a sacrifice of otherwise saved code size, but it just seems a bit overboard.
But note that with your latest commit, things are far worse,
would be nice if someone could have let me know
you didn't provide any tests
I was waiting for a sign that anything I was doing had even a chance of being accepted, something like "ok, everything looks good, now we need ____", before I spend significantly more time on the finishing requirements other than what downstream projects need, including documentation. The age of this thread was kept in mind...
That's just your opinion
without understanding the difference or having explanations to consider, opinions are all that remains
I am more interested in finding a common ground that we can all use
as was I, and still am. looking forward to trying it and working with you despite mixed feelings so far
Here are my reproductions steps, where I am using the hyperfine tool to benchmark Ninja with and without a jobserver pool:
# Clone superbuild repository.
git clone https://gitlab.kitware.com/paraview/common-superbuild.git
# Configure a self-test build with a few projects with CMake in the `_build` directory.cd common-superbuild/selftest
cmake -B _build -S. \
-DENABLE_zlib=1 \
-DENABLE_zstd=1 \
-DENABLE_szip=1 \
-DENABLE_snappy=1 \
-DENABLE_ninja=1 \
-DENABLE_jsoncpp=1 \
-DENABLE_freetype=1 \
-DENABLE_imath=1 \
-DENABLE_bzip2=1 \
-DENABLE_lz4=1 \
-GNinja
# Run Ninja to download and extract all project sources (this can be *long* and not parallel)
/usr/bin/time ninja -C _build download-all
# There is no configure-all target, so let's do it manually.# This is also *long* and not parallel.# NOTE: Adding certain projects (e.g. `llvm` or `meson`) requires building + installing# python3, as well as all its dependencies, during their configure step. This skews benchmarked# numbers tremendously, so avoid them if possible, or just disable the following two# lines (benchmarks will include the times for the long on-parallel `configure` steps though :-()
CONFIGURE_TARGETS=$(ninja -C _build -t inputs all | grep -e '-configure$')
/usr/bin/time ninja -C _build ${CONFIGURE_TARGETS}# Rename _build to _build0, so we can reuse its content in multiple builds.# Unfortunately, `ninja -C _build clean` or `ninja -C _build -t clean` do not clean properly.
mv _build _build0
# Now benchmark Ninja without and with --jobserver# The preparation step restores the _build directory to its pristine configured state.
hyperfine --prepare "rm -rf _build && cp -rp _build0 _build" \
"ninja -C _build" \
"ninja -C _build --jobserver"
At first, naively running this on my powerful work machine doesn't show any significant difference:
# Running on a machine with 96 cores: no meaningfull difference
Benchmark 1: ninja -C _build
Time (mean ± σ): 7.751 s ± 0.092 s [User: 73.560 s, System: 3.978 s]
Range (min … max): 7.615 s … 7.904 s 10 runs
Benchmark 2: ninja -C _build --jobserver
Time (mean ± σ): 7.743 s ± 0.070 s [User: 73.627 s, System: 3.832 s]
Range (min … max): 7.621 s … 7.826 s 10 runs
Summary
ninja -C _build --jobserver ran
1.00 ± 0.01 times faster than ninja -C _build
But this is not surprising. This build is simply not very parallel and can use all the CPU cores it needs on this machine, with or without a jobserver. Hence the same overall build times.
The interesting part is using Linux cgroups to restrict the number of CPU cores available to the commands, and here we say a difference. I am on a Debian system and using systemd-run --scope --property AllowedCPUs=XX-YY <command> to run hyperfine, where XX and YY are the starting and ending CPU indices.
When using 8 cores only (e.g. AllowedCPUs=0-7), the --jobserver version is x1.18 faster.
When using 4 cores only (e.g. AllowedCPUs=0-3), the --jobserver version is x1.06 faster only.
Here are the benchmark results for the 8-core case:
# Running in a Linux cgroups with only 8 CPUs (systemd-run --scope --property AllowedCPUs=0-7 hyperfine ...)
Benchmark 1: ninja -C _build
Time (mean ± σ): 13.263 s ± 0.198 s [User: 68.994 s, System: 2.825 s]
Range (min … max): 12.900 s … 13.576 s 10 runs
Benchmark 2: ninja -C _build --jobserver
Time (mean ± σ): 11.283 s ± 1.040 s [User: 64.962 s, System: 2.887 s]
Range (min … max): 10.176 s … 12.802 s 10 runs
Summary
ninja -C _build --jobserver ran
1.18 ± 0.11 times faster than ninja -C _build
And for the 4-core case:
# Running in a Linux cgroups with only 4 CPUs (systemd-run --scope --property AllowedCPUs=0-3 hyperfine ...)
Benchmark 1: ninja -C _build
Time (mean ± σ): 20.981 s ± 0.217 s [User: 68.251 s, System: 2.707 s]
Range (min … max): 20.614 s … 21.396 s 10 runs
Benchmark 2: ninja -C _build --jobserver
Time (mean ± σ): 19.728 s ± 1.111 s [User: 64.685 s, System: 2.815 s]
Range (min … max): 17.838 s … 21.483 s 10 runs
Summary
ninja -C _build --jobserver ran
1.06 ± 0.06 times faster than ninja -C _build
Notice however that the volatility of the jobserver builds in the previous section is high, but I can reproduce the same mean over several hyperfine runs locally :-/
Now, I also wanted to get a better understanding of what's going on, and also try to get something that is more parallel, so I ended up creating my own tiny super-build system. Its main benefit is the ability to generate a build_trace.json file that describes the execution of all build tasks for analysis.
Now for the bad news: I can see a jobserver-related reproducible performance regression with this new build system.
Similarly to previously, running with all cores shows no significant difference (see below).
However, when limiting to 8 CPUs, the jobserver build is noticeably slower (x1.08, with high volatility, but that can be reproduced consistently).
# Running with full 96 cores
Benchmark 1: ninja
Time (mean ± σ): 8.963 s ± 0.168 s [User: 247.292 s, System: 14.446 s]
Range (min … max): 8.677 s … 9.233 s 10 runs
Benchmark 2: ninja --jobserver
Time (mean ± σ): 8.897 s ± 0.197 s [User: 257.761 s, System: 15.928 s]
Range (min … max): 8.666 s … 9.341 s 10 runs
Summary
ninja --jobserver ran
1.01 ± 0.03 times faster than ninja
# Limited to 8 cores
Benchmark 1: ninja
Time (mean ± σ): 22.953 s ± 0.395 s [User: 160.829 s, System: 7.868 s]
Range (min … max): 22.494 s … 23.702 s 10 runs
Benchmark 2: ninja --jobserver
Time (mean ± σ): 24.821 s ± 1.531 s [User: 152.558 s, System: 8.320 s]
Range (min … max): 20.797 s … 25.800 s 10 runs
Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
Summary
ninja ran
1.08 ± 0.07 times faster than ninja --jobserver
@digit-google After experimenting today, I have good and bad news :-)
While those numbers are interesting, IMHO your test case does not really represent why people need jobserver support in ninja.
There are multi-level build systems out there that blindly translate "component sub-build invocation" into a new independent parallel build, e.g. make -j4. AFAIR kati did this for every $(MAKE) inside an Android build. So with N components you end up with 4*N parallel build steps, leading to a complete overload of the build machine.
The final goal is that the top-level build tool provides a job server with T tokens and any build tool instance, including the top-level instance, adhere to the limit, i.e. in total there are never more than T build steps running in parallel (not counting the build tool instances itself, because each of them should be waiting for their children to complete).
I think I have an explanation for this phenomenom though.
As far as I know, none of the proposed implementations changes the fact that Ninja only decides to launch new commands when at least one of them has completed. In other words, if a job slot becomes available in the pool, Ninja will not see it until the end of one of its current running subprocesses, even if it has capacity to launch more of them.
(This is not how GNU Make works by the way, it is capable of waiting for both process completion and new token in the FIFO at the same time, at least on Posix, not sure about Windows).
Here's an example to illustrate this. Let's imagine a top-level build that invokes two sub-builds, each one of them has two parallel tasks, but one of them has short ones, and the other has long ones. If there are enough cores for everyone, everything can be dispatched in parallel, which would look like:
Now let's imagine that there are only 2 cores available on the machine this runs on, without a jobserver, 4 processes are started at the same time, and the OS will control parallelism by prempting them as it see fit. Assuming these are purely CPU-bound tasks, the most optimal scheduling would end up with a build time that is similar to this timeline:
Now, if a jobserver is used to synchronize the parallelism, it is possible to end up in a situation where each sub-build only gets one token initially:
[short1]
[long___________________1]
The first sub-build completes, and reuses its token to finish its second task:
[short1][short2]
[long___________________1]
At the end of [short2] the first sub-build releases its token to the pool, but the second sub-build doesn't know about it, and waits for the completion of its first task, before launching the second one:
And a long build time is the result, due to poor scheduling.
It would be better if Ninja could detect the new token immediately to launch the second long task asap, as in:
Some feedback, since I could roll this feature in the Ninja binary used by the Fuchsia build, and changed our builder configurations to use it.
The Fuchsia CI bots build around 90 different build configurations of the project, only a few of them actually launch Ninja sub-builds from the top-level one (to generate SDKs that require binaries from different CPU architectures and API levels, then merging the results into a final SDK archive, in case you're curious :-))
I could enable the jobserver feature for all build configurations and perform clean build time comparisons:
One SDK build configuration saves 12 minutes (from 1h to 48 minutes) due to it.
Other SDK build configurations save only 6 minutes (e.g. from 42 to 36 minutes), which is still a significant win.
Other build configurations, which do not use sub-builds, are simply not affected (as expected, but it's still nice to check).
These confirm the savings I could observe locally on my own workstation (these are builds running on CI bots of various CPU/RAM dimensions, with remote builds enabled to boot, so we run everything with -j CPU_COUNT*10 on these).
The jobserver feature is now enabled by default on all Fuchsia SDK build configurations.
I addressed the few review comments on the PR #2506 that were sent, and would welcome any more feedback.
I would be grateful if anyone could use the latest binaries generated by the Github CI and see if this improves their own multi-build times, or report issues they are facing.
Regarding the sub-optimal scheduling described in my previous comment. Solving this will require changing the way Subprocess::DoWork() works in order to either detect new types of events (and report them). I think this is better left for another PR and discussion. What do you think?
I would be grateful if anyone could use the latest binaries generated by the Github CI and see if this improves their own multi-build times, or report issues they are facing.
I've been using it for the last week or so. The only quirk was the interaction between NINJA_JOBSERVER=1 and --jobserver, but that wasn't a big deal at all, I just hadn't clocked it at first. I've been able to increase parallelism and not have LTO bomb the system on large world rebuilds.
I think this is better left for another PR and discussion. What do you think?
Yeah, I'd hate to see it delayed further, especially given nobody so far (hopefully more can test) have raised issues with the changes. It also means that we're not fighting for interest/attention on a topic that deserves its own space, I think?
Reacted by Jean-Christophe Fillion-Robin, Ben Boeckel, David Fong, Wilfried Goesgens, Randy MacLeod, Andreas Gnau, Andrew Gunnerson, Sam James, Kesavan Yogeswaran, stephan-cr and 5 more
Activity
Add GNU make jobserver client support
stefanb2 commentedon Apr 27, 2016
I have tested this implementation over the last few weeks in two different recursive GNU make based build systems that originally had M+1 GNU make instances:
FYI: google/kati was used to convert existing single makefile GNU make parts to Ninja build file.
nico commentedon Apr 27, 2016
Thanks for the patch!
We've discussed this on the mailing list a few times (e.g. here https://groups.google.com/forum/#!searchin/ninja-build/jobserver/ninja-build/PUlsr7-jpI0/Ga19TOg1c14J). Ninja works best if it knows about the whole build. Now that kati exists, one can convert those to ninja files and munge them up to have a single build manifest (that's Android's transition strategy from Make to Ninja -- they use kati to get everything converted to Ninja files, and then they're incrementally converting directories to use something-not-make -- and then kati produces parts of their Ninja files and the new thing produces parts of the ninja files.)
Is your use case that you have recursive makefiles?
stefanb2 commentedon Apr 27, 2016
I could have guessed that this has been discussed before, because I'm surely not the first person facing such a situation.
Here are my reasons for requesting this:
IMHO my patch provides a good solution, considering
Add GNU make jobserver client support
Add GNU make jobserver client support
Add GNU make jobserver client support
Add GNU make jobserver client support
Add GNU make jobserver client support
Add GNU make jobserver client support
Add GNU make jobserver client support
Add GNU make jobserver client support
maximuska commentedon Aug 7, 2016
Another possible reason for having jobserver in ninja seems to be LTO support in gcc. -flto=jobserver tells gcc to use GNU make's job server mode to determine the number of parallel jobs. The alternative is to spawn a fixed number of jobs with e.g., -flto=16.
fabio-porcedda commentedon Mar 10, 2017
I would like too have this feature merged, i simply cannot convert all projects to ninja-build because i'm not allowed to do that.
@stefanb2 Thanks a lot for your work
123 remaining items
mathstuf commentedon Oct 1, 2024
This project is a collection of inner projects driven by an outer project: https://gitlab.kitware.com/paraview/common-superbuild
You'll want to use the
selftestsubdirectory and to enable some selection of projects (the compression libraries are probably a good candidate set). To get mismatched generators (i.e., different generators for the "superbuild" compared to the inner builds, specifyCMAKE_GENERATOR "inner name"in the_superbuild_ExternalProject_addcall incmake/SuperbuildMacros.cmake(line 1550 or so). There's also some logic that may need tweaked to understand thatninjacan also provide a jobserver around line 30 incmake/SuperbuildExternalProject.cmake.digit-google commentedon Oct 1, 2024
That's what I tried initially with your PR, but the resulting binary didn't provide any improvements to my build, except very reduced volatility. Here are the corresponding raw numbers I had at the time for a tiny Fuchsia build, in case you are interested:
But note that with your latest commit, things are far worse, unfortunately. You didn't even explain why you decided to change the capacity logic, nor provide any meaningful numbers for its usefulness. And again, you didn't provide any tests verifying the basic correctness of your code.
I know you have worked considerably on this, and addressed a lot of review comments, but you didn't address all the issues/ The sad truth is that this was still not good enough, even performance wise, at least for my use case. Hence the motivation to start something else.
That's just your opinion.
At this point, I am more interested in finding a common ground that we can all use to verify that whatever lands into the final Ninja tree works correctly for everyone.
Because PR#2506 works for me doesn't mean it would work for you or other people, hence why it is so important to get good common regression tests. here. And fix whatever needs to be fixed otherwise.
mcprat commentedon Oct 1, 2024
Ah ok... so there is much more going on here...
I was following the initial complaint or concern about the value being "infinite", so I tried to determine a working finite value, or at least I thought it was working, that coincides with the whole reason that the "capacity" value was set up in the first place in 8a2575e, in order to gently reach maximum parallelism without an instantaneous load which can shock the system. What you're calling "volatility" in this exact case may be on purpose... or am I misunderstanding something?
The value I put is supposed to represent a capacity of
1 + 2^nwithnincrementing for every loop of the builder per finished edge until the next step is the maximum. Because my ability to test high parallelism is limited, it could be that it's actually1 + 2*ninstead...So do you know which optimization is improving the performance and where? I would assume it has something to do with how the "acquire" and "release" functions handle the data. I would like to learn something and be corrected and informed instead of just bypassed. Perhaps extreme optimizations are necessary in one or a few spots, but it would be nice if we can determine where it actually matters so other spots in the code can return to simplicity. I understand that in the world of optimization, speed requires a sacrifice of otherwise saved code size, but it just seems a bit overboard.
would be nice if someone could have let me know
I was waiting for a sign that anything I was doing had even a chance of being accepted, something like "ok, everything looks good, now we need ____", before I spend significantly more time on the finishing requirements other than what downstream projects need, including documentation. The age of this thread was kept in mind...
without understanding the difference or having explanations to consider, opinions are all that remains
as was I, and still am. looking forward to trying it and working with you despite mixed feelings so far
digit-google commentedon Oct 2, 2024
After experimenting today, I have good and bad news :-)
The good news is that thanks to @mathstuf , I have been able to experiment with https://gitlab.kitware.com/paraview/common-superbuild which shows a deterministic though modest improvement when
--jobserveris being used.Here are my reproductions steps, where I am using the
hyperfinetool to benchmark Ninja with and without a jobserver pool:At first, naively running this on my powerful work machine doesn't show any significant difference:
But this is not surprising. This build is simply not very parallel and can use all the CPU cores it needs on this machine, with or without a jobserver. Hence the same overall build times.
The interesting part is using Linux cgroups to restrict the number of CPU cores available to the commands, and here we say a difference. I am on a Debian system and using
systemd-run --scope --property AllowedCPUs=XX-YY <command>to run hyperfine, whereXXandYYare the starting and ending CPU indices.When using 8 cores only (e.g.
AllowedCPUs=0-7), the--jobserverversion is x1.18 faster.When using 4 cores only (e.g.
AllowedCPUs=0-3), the--jobserverversion is x1.06 faster only.Here are the benchmark results for the 8-core case:
And for the 4-core case:
That's for the good news
digit-google commentedon Oct 2, 2024
Notice however that the volatility of the jobserver builds in the previous section is high, but I can reproduce the same mean over several hyperfine runs locally :-/
Now, I also wanted to get a better understanding of what's going on, and also try to get something that is more parallel, so I ended up creating my own tiny super-build system. Its main benefit is the ability to generate a
build_trace.jsonfile that describes the execution of all build tasks for analysis.You can find it at https://github.com/digit-google/ninja-recursive-build, the
README.mdcontains all instructions you need to replicate my results and play with it.Now for the bad news: I can see a jobserver-related reproducible performance regression with this new build system.
Similarly to previously, running with all cores shows no significant difference (see below).
However, when limiting to 8 CPUs, the jobserver build is noticeably slower (x1.08, with high volatility, but that can be reproduced consistently).
stefanb2 commentedon Oct 2, 2024
While those numbers are interesting, IMHO your test case does not really represent why people need jobserver support in ninja.
There are multi-level build systems out there that blindly translate "component sub-build invocation" into a new independent parallel build, e.g.
make -j4. AFAIRkatidid this for every$(MAKE)inside an Android build. So with N components you end up with 4*N parallel build steps, leading to a complete overload of the build machine.The final goal is that the top-level build tool provides a job server with T tokens and any build tool instance, including the top-level instance, adhere to the limit, i.e. in total there are never more than T build steps running in parallel (not counting the build tool instances itself, because each of them should be waiting for their children to complete).
digit-google commentedon Oct 2, 2024
I think I have an explanation for this phenomenom though.
As far as I know, none of the proposed implementations changes the fact that Ninja only decides to launch new commands when at least one of them has completed. In other words, if a job slot becomes available in the pool, Ninja will not see it until the end of one of its current running subprocesses, even if it has capacity to launch more of them.
(This is not how GNU Make works by the way, it is capable of waiting for both process completion and new token in the FIFO at the same time, at least on Posix, not sure about Windows).
Here's an example to illustrate this. Let's imagine a top-level build that invokes two sub-builds, each one of them has two parallel tasks, but one of them has short ones, and the other has long ones. If there are enough cores for everyone, everything can be dispatched in parallel, which would look like:
Which can be summarized as:
Now let's imagine that there are only 2 cores available on the machine this runs on, without a jobserver, 4 processes are started at the same time, and the OS will control parallelism by prempting them as it see fit. Assuming these are purely CPU-bound tasks, the most optimal scheduling would end up with a build time that is similar to this timeline:
Now, if a jobserver is used to synchronize the parallelism, it is possible to end up in a situation where each sub-build only gets one token initially:
The first sub-build completes, and reuses its token to finish its second task:
At the end of
[short2]the first sub-build releases its token to the pool, but the second sub-build doesn't know about it, and waits for the completion of its first task, before launching the second one:And a long build time is the result, due to poor scheduling.
It would be better if Ninja could detect the new token immediately to launch the second long task asap, as in:
This would end up with a better build time, even though not the optimal one that can be achieved without a jobserver.
digit-google commentedon Oct 16, 2024
Some feedback, since I could roll this feature in the Ninja binary used by the Fuchsia build, and changed our builder configurations to use it.
The Fuchsia CI bots build around 90 different build configurations of the project, only a few of them actually launch Ninja sub-builds from the top-level one (to generate SDKs that require binaries from different CPU architectures and API levels, then merging the results into a final SDK archive, in case you're curious :-))
I could enable the jobserver feature for all build configurations and perform clean build time comparisons:
These confirm the savings I could observe locally on my own workstation (these are builds running on CI bots of various CPU/RAM dimensions, with remote builds enabled to boot, so we run everything with
-j CPU_COUNT*10on these).The jobserver feature is now enabled by default on all Fuchsia SDK build configurations.
I addressed the few review comments on the PR #2506 that were sent, and would welcome any more feedback.
I would be grateful if anyone could use the latest binaries generated by the Github CI and see if this improves their own multi-build times, or report issues they are facing.
Regarding the sub-optimal scheduling described in my previous comment. Solving this will require changing the way
Subprocess::DoWork()works in order to either detect new types of events (and report them). I think this is better left for another PR and discussion. What do you think?thesamesam commentedon Oct 17, 2024
I've been using it for the last week or so. The only quirk was the interaction between
NINJA_JOBSERVER=1and--jobserver, but that wasn't a big deal at all, I just hadn't clocked it at first. I've been able to increase parallelism and not have LTO bomb the system on large world rebuilds.Yeah, I'd hate to see it delayed further, especially given nobody so far (hopefully more can test) have raised issues with the changes. It also means that we're not fighting for interest/attention on a topic that deserves its own space, I think?
contrib: add MESONBUILD similar to CMAKEBUILD
jhasse commentedon May 17, 2025
Thanks to @stefanb2, @hundeboll, @mcprat, @digit-google and everyone else who helped over the years :)
Revert "sanitycheck: Default to using Ninja"