Skip to content

Add GNU make jobserver client support #1139

Closed
Closed

Description

@stefanb2

As long as ninja is the only build execution tool, the current ninja -jN implementation works fine.

But when you try to convert parts of an existing recursive GNU make based SW build system to ninja, then you have the following situation:

  • top-level GNU Make (with -jX, acts as job server)
  • M instances of GNU make (with -j, act as job server clients)
  • N instances of ninja (don't know anything about job server)

Simply calling `ninja -jY' isn't enough, because then the ninja instances will try to run Y*N jobs, plus the X jobs from the GNU make instances, causing the build host to overload. Relying on -lZ to fix this issue is sub-optimal, because load average is sometimes too slow to reflect the actual situation on the build host.

It would be nice if GNU make jobserver client support could be added to Ninja. Then the N ninja instances would cooperate with the M GNU make instances and on the build host only X jobs would be executed at one time.

Activity

  1. added a commit that references this issue on Apr 27, 2016
  2. stefanb2 commented on Apr 27, 2016

    @stefanb2
    ContributorAuthor

    I have tested this implementation over the last few weeks in two different recursive GNU make based build systems that originally had M+1 GNU make instances:

    • use case A: top-level GNU make, 1 ninja instance, M-1 GNU make instances
    • use case B: top-level GNU make, N ninja instances, M-N GNU make instances

    FYI: google/kati was used to convert existing single makefile GNU make parts to Ninja build file.

  3. nico commented on Apr 27, 2016

    @nico
    Collaborator

    Thanks for the patch!

    We've discussed this on the mailing list a few times (e.g. here https://groups.google.com/forum/#!searchin/ninja-build/jobserver/ninja-build/PUlsr7-jpI0/Ga19TOg1c14J). Ninja works best if it knows about the whole build. Now that kati exists, one can convert those to ninja files and munge them up to have a single build manifest (that's Android's transition strategy from Make to Ninja -- they use kati to get everything converted to Ninja files, and then they're incrementally converting directories to use something-not-make -- and then kati produces parts of their Ninja files and the new thing produces parts of the ninja files.)

    Is your use case that you have recursive makefiles?

  4. stefanb2 commented on Apr 27, 2016

    @stefanb2
    ContributorAuthor

    I could have guessed that this has been discussed before, because I'm surely not the first person facing such a situation.

    Here are my reasons for requesting this:

    1. recursion: kati currently can't translate recursive GNU make based build systems, like Linux kernel kbuild. IMHO a major effort and unfortunately I can't wait for kati to provide this, hence the such sub-component builds will have to stay with GNU make for the time being.
    2. missing features: kati currently can't translate fully modularized GNU make based build systems, i.e. where each component is built in isolation and in a separate build directory, so that all ninja.build files could be merged into a single one. While IMHO not such a major issue as (1) it is much simpler to replace the lowest-level $(MAKE) recipe with a kati/ninja recipe. Parsing + merging might also introduce unnecessary build delay (needs to be seen what would happen in real life)
    3. technical barriers: e.g. sub-component builds that run behind a "chroot firewall". Even if everything moves to Ninja, you would still need 1 (main) + N (one for each chroot) ninja instances that need to cooperate. Ninja doesn't offer anything like that.
    4. too simple workarounds: AOSP makeparallel + kati/ninja runs all $(MAKE) instances hard-coded with "make -j4" with no cooperation between any of the GNU make instances. That is only acceptable if you have no or only a few or small $(MAKE) invocations from the ninja.build file.
    5. organizational barriers: even if it might be possible to use kati/ninja to convert an existing GNU make base sub-part of the system, you might not be allowed to do so. Such sub-component builds need to stay with GNU make.
    6. You ask: why not split the build up and run them as separate builds? Goto (5)...

    IMHO my patch provides a good solution, considering

    • how small the required changes to ninja are,
    • that the default behaviour is completely unchanged, and
    • that this will make the life easier for many other ninja users which face the same issues
  5. maximuska commented on Aug 7, 2016

    @maximuska
    Contributor

    Another possible reason for having jobserver in ninja seems to be LTO support in gcc. -flto=jobserver tells gcc to use GNU make's job server mode to determine the number of parallel jobs. The alternative is to spawn a fixed number of jobs with e.g., -flto=16.

  6. fabio-porcedda commented on Mar 10, 2017

    @fabio-porcedda

    I would like too have this feature merged, i simply cannot convert all projects to ninja-build because i'm not allowed to do that.

    @stefanb2 Thanks a lot for your work

  7. 123 remaining items

  8. mathstuf commented on Oct 1, 2024

    @mathstuf
    Contributor

    Any suggestions for such a multi-build? Any recursive Ninja invocations (either from Meson or CMake) would fit the build I think. Or maybe something that uses Make at the top-level, but then builds multiple things with Ninja.

    This project is a collection of inner projects driven by an outer project: https://gitlab.kitware.com/paraview/common-superbuild

    You'll want to use the selftest subdirectory and to enable some selection of projects (the compression libraries are probably a good candidate set). To get mismatched generators (i.e., different generators for the "superbuild" compared to the inner builds, specify CMAKE_GENERATOR "inner name" in the _superbuild_ExternalProject_add call in cmake/SuperbuildMacros.cmake (line 1550 or so). There's also some logic that may need tweaked to understand that ninja can also provide a jobserver around line 30 in cmake/SuperbuildExternalProject.cmake.

  9. digit-google commented on Oct 1, 2024

    @digit-google
    Contributor

    so this "degraded performance"... is it just that I came up with a scaling method for capacity instead of "infinite"? then the performance would be the same if I put it back to infinite perhaps?

    That's what I tried initially with your PR, but the resulting binary didn't provide any improvements to my build, except very reduced volatility. Here are the corresponding raw numbers I had at the time for a tiny Fuchsia build, in case you are interested:

    ninja-upstream & -j128
    
    4:35.20elapsed 8249%CPU
    4:45.41elapsed 7914%CPU
    4:46.71elapsed 7957%CPU
    4:49.69elapsed 7829%CPU
    5:01.97elapsed 7394%CPU
    
    ninja-digit & jobserver_pool.py --token-count=128
    
    3:55.67elapsed 10459%CPU
    3:59.62elapsed 10247%CPU
    4:01.02elapsed 10235%CPU
    4:05.92elapsed 9984%CPU
    4:17.52elapsed 9524%CPU
    
    jobserver-final & jobserver_pool.py --token-count=j128
    
    4:47.13elapsed 8187%CPU
    4:47.43elapsed 8130%CPU
    4:47.79elapsed 8153%CPU
    4:48.15elapsed 8134%CPU
    4:48.28elapsed 8204%CPU
    

    But note that with your latest commit, things are far worse, unfortunately. You didn't even explain why you decided to change the capacity logic, nor provide any meaningful numbers for its usefulness. And again, you didn't provide any tests verifying the basic correctness of your code.

    I know you have worked considerably on this, and addressed a lot of review comments, but you didn't address all the issues/ The sad truth is that this was still not good enough, even performance wise, at least for my use case. Hence the motivation to start something else.

    maybe I should have mentioned my system has only 4 cores

    the jobserver protocol is supposed to be simple but what I'm seeing instead is as verbose and unreadable as you can possibly make it, pulling out all the tricks you have ever learned, for what benefit honestly?

    don't get me wrong, I'm sure it works great, it just looks vastly over-engineered at first glance

    That's just your opinion.

    At this point, I am more interested in finding a common ground that we can all use to verify that whatever lands into the final Ninja tree works correctly for everyone.

    Because PR#2506 works for me doesn't mean it would work for you or other people, hence why it is so important to get good common regression tests. here. And fix whatever needs to be fixed otherwise.

  10. mcprat commented on Oct 1, 2024

    @mcprat
    Contributor

    That's what I tried initially with your PR, but the resulting binary didn't provide any improvements to my build, except very reduced volatility

    Ah ok... so there is much more going on here...

    You didn't even explain why you decided to change the capacity logic, nor provide any meaningful numbers for its usefulness

    I was following the initial complaint or concern about the value being "infinite", so I tried to determine a working finite value, or at least I thought it was working, that coincides with the whole reason that the "capacity" value was set up in the first place in 8a2575e, in order to gently reach maximum parallelism without an instantaneous load which can shock the system. What you're calling "volatility" in this exact case may be on purpose... or am I misunderstanding something?

    The value I put is supposed to represent a capacity of 1 + 2^n with n incrementing for every loop of the builder per finished edge until the next step is the maximum. Because my ability to test high parallelism is limited, it could be that it's actually 1 + 2*n instead...

    So do you know which optimization is improving the performance and where? I would assume it has something to do with how the "acquire" and "release" functions handle the data. I would like to learn something and be corrected and informed instead of just bypassed. Perhaps extreme optimizations are necessary in one or a few spots, but it would be nice if we can determine where it actually matters so other spots in the code can return to simplicity. I understand that in the world of optimization, speed requires a sacrifice of otherwise saved code size, but it just seems a bit overboard.

    But note that with your latest commit, things are far worse,

    would be nice if someone could have let me know

    you didn't provide any tests

    I was waiting for a sign that anything I was doing had even a chance of being accepted, something like "ok, everything looks good, now we need ____", before I spend significantly more time on the finishing requirements other than what downstream projects need, including documentation. The age of this thread was kept in mind...

    That's just your opinion

    without understanding the difference or having explanations to consider, opinions are all that remains

    I am more interested in finding a common ground that we can all use

    as was I, and still am. looking forward to trying it and working with you despite mixed feelings so far

  11. digit-google commented on Oct 2, 2024

    @digit-google
    Contributor

    After experimenting today, I have good and bad news :-)

    The good news is that thanks to @mathstuf , I have been able to experiment with https://gitlab.kitware.com/paraview/common-superbuild which shows a deterministic though modest improvement when --jobserver is being used.

    Here are my reproductions steps, where I am using the hyperfine tool to benchmark Ninja with and without a jobserver pool:

    # Clone superbuild repository.
    git clone https://gitlab.kitware.com/paraview/common-superbuild.git
    
    # Configure a self-test build with a few projects with CMake in the `_build` directory.
    cd common-superbuild/selftest
    
    cmake -B _build -S. \
      -DENABLE_zlib=1 \
      -DENABLE_zstd=1 \
      -DENABLE_szip=1 \
      -DENABLE_snappy=1 \
      -DENABLE_ninja=1 \
      -DENABLE_jsoncpp=1 \
      -DENABLE_freetype=1 \
      -DENABLE_imath=1 \
      -DENABLE_bzip2=1 \
      -DENABLE_lz4=1 \
      -GNinja
    
    # Run Ninja to download and extract all project sources (this can be *long* and not parallel)
    /usr/bin/time ninja -C _build download-all
    
    # There is no configure-all target, so let's do it manually.
    # This is also *long* and not parallel.
    # NOTE: Adding certain projects (e.g. `llvm` or `meson`) requires building + installing
    # python3, as well as all its dependencies, during their configure step. This skews benchmarked
    # numbers tremendously, so avoid them if possible, or just disable the following two
    # lines (benchmarks will include the times for the long on-parallel `configure` steps though :-()
    CONFIGURE_TARGETS=$(ninja -C _build -t inputs all | grep -e '-configure$')
    /usr/bin/time ninja -C _build ${CONFIGURE_TARGETS}
    
    # Rename _build to _build0, so we can reuse its content in multiple builds.
    # Unfortunately, `ninja -C _build clean` or `ninja -C _build -t clean` do not clean properly.
    mv _build _build0
    
    # Now benchmark Ninja without and with --jobserver
    # The preparation step restores the _build directory to its pristine configured state.
    hyperfine --prepare "rm -rf _build && cp -rp _build0 _build" \
      "ninja -C _build" \
      "ninja -C _build --jobserver"

    At first, naively running this on my powerful work machine doesn't show any significant difference:

    # Running on a machine with 96 cores: no meaningfull difference
    Benchmark 1: ninja -C _build
      Time (mean ± σ):      7.751 s ±  0.092 s    [User: 73.560 s, System: 3.978 s]
      Range (min … max):    7.615 s …  7.904 s    10 runs
    
    Benchmark 2: ninja -C _build --jobserver
      Time (mean ± σ):      7.743 s ±  0.070 s    [User: 73.627 s, System: 3.832 s]
      Range (min … max):    7.621 s …  7.826 s    10 runs
    
    Summary
      ninja -C _build --jobserver ran
        1.00 ± 0.01 times faster than ninja -C _build
    

    But this is not surprising. This build is simply not very parallel and can use all the CPU cores it needs on this machine, with or without a jobserver. Hence the same overall build times.

    The interesting part is using Linux cgroups to restrict the number of CPU cores available to the commands, and here we say a difference. I am on a Debian system and using systemd-run --scope --property AllowedCPUs=XX-YY <command> to run hyperfine, where XX and YY are the starting and ending CPU indices.

    When using 8 cores only (e.g. AllowedCPUs=0-7), the --jobserver version is x1.18 faster.

    When using 4 cores only (e.g. AllowedCPUs=0-3), the --jobserver version is x1.06 faster only.

    Here are the benchmark results for the 8-core case:

    # Running in a Linux cgroups with only 8 CPUs (systemd-run --scope --property AllowedCPUs=0-7 hyperfine ...)
    Benchmark 1: ninja -C _build
      Time (mean ± σ):     13.263 s ±  0.198 s    [User: 68.994 s, System: 2.825 s]
      Range (min … max):   12.900 s … 13.576 s    10 runs
     
    Benchmark 2: ninja -C _build --jobserver
      Time (mean ± σ):     11.283 s ±  1.040 s    [User: 64.962 s, System: 2.887 s]
      Range (min … max):   10.176 s … 12.802 s    10 runs
     
    Summary
      ninja -C _build --jobserver ran
        1.18 ± 0.11 times faster than ninja -C _build
    

    And for the 4-core case:

    # Running in a Linux cgroups with only 4 CPUs (systemd-run --scope --property AllowedCPUs=0-3 hyperfine ...)
    Benchmark 1: ninja -C _build
      Time (mean ± σ):     20.981 s ±  0.217 s    [User: 68.251 s, System: 2.707 s]
      Range (min … max):   20.614 s … 21.396 s    10 runs
    
    Benchmark 2: ninja -C _build --jobserver
      Time (mean ± σ):     19.728 s ±  1.111 s    [User: 64.685 s, System: 2.815 s]
      Range (min … max):   17.838 s … 21.483 s    10 runs
    
    Summary
      ninja -C _build --jobserver ran
        1.06 ± 0.06 times faster than ninja -C _build
    

    That's for the good news

  12. digit-google commented on Oct 2, 2024

    @digit-google
    Contributor

    Notice however that the volatility of the jobserver builds in the previous section is high, but I can reproduce the same mean over several hyperfine runs locally :-/

    Now, I also wanted to get a better understanding of what's going on, and also try to get something that is more parallel, so I ended up creating my own tiny super-build system. Its main benefit is the ability to generate a build_trace.json file that describes the execution of all build tasks for analysis.

    You can find it at https://github.com/digit-google/ninja-recursive-build, the README.md contains all instructions you need to replicate my results and play with it.

    Now for the bad news: I can see a jobserver-related reproducible performance regression with this new build system.

    Similarly to previously, running with all cores shows no significant difference (see below).
    However, when limiting to 8 CPUs, the jobserver build is noticeably slower (x1.08, with high volatility, but that can be reproduced consistently).

    # Running with full 96 cores
    Benchmark 1: ninja
      Time (mean ± σ):      8.963 s ±  0.168 s    [User: 247.292 s, System: 14.446 s]
      Range (min … max):    8.677 s …  9.233 s    10 runs
     
    Benchmark 2: ninja --jobserver
      Time (mean ± σ):      8.897 s ±  0.197 s    [User: 257.761 s, System: 15.928 s]
      Range (min … max):    8.666 s …  9.341 s    10 runs
     
    Summary
      ninja --jobserver ran
        1.01 ± 0.03 times faster than ninja
    
    # Limited to 8 cores
    Benchmark 1: ninja
      Time (mean ± σ):     22.953 s ±  0.395 s    [User: 160.829 s, System: 7.868 s]
      Range (min … max):   22.494 s … 23.702 s    10 runs
     
    Benchmark 2: ninja --jobserver
      Time (mean ± σ):     24.821 s ±  1.531 s    [User: 152.558 s, System: 8.320 s]
      Range (min … max):   20.797 s … 25.800 s    10 runs
     
      Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
     
    Summary
      ninja ran
        1.08 ± 0.07 times faster than ninja --jobserver
    
  13. stefanb2 commented on Oct 2, 2024

    @stefanb2
    ContributorAuthor

    @digit-google After experimenting today, I have good and bad news :-)

    While those numbers are interesting, IMHO your test case does not really represent why people need jobserver support in ninja.

    There are multi-level build systems out there that blindly translate "component sub-build invocation" into a new independent parallel build, e.g. make -j4. AFAIR kati did this for every $(MAKE) inside an Android build. So with N components you end up with 4*N parallel build steps, leading to a complete overload of the build machine.

    The final goal is that the top-level build tool provides a job server with T tokens and any build tool instance, including the top-level instance, adhere to the limit, i.e. in total there are never more than T build steps running in parallel (not counting the build tool instances itself, because each of them should be waiting for their children to complete).

  14. digit-google commented on Oct 2, 2024

    @digit-google
    Contributor

    I think I have an explanation for this phenomenom though.

    As far as I know, none of the proposed implementations changes the fact that Ninja only decides to launch new commands when at least one of them has completed. In other words, if a job slot becomes available in the pool, Ninja will not see it until the end of one of its current running subprocesses, even if it has capacity to launch more of them.

    (This is not how GNU Make works by the way, it is capable of waiting for both process completion and new token in the FIFO at the same time, at least on Posix, not sure about Windows).

    Here's an example to illustrate this. Let's imagine a top-level build that invokes two sub-builds, each one of them has two parallel tasks, but one of them has short ones, and the other has long ones. If there are enough cores for everyone, everything can be dispatched in parallel, which would look like:

    top-level
    |___subbuild_A
    |   |_____[short1]
    |   |_____[short2]
    |
    |___subbuild_B
        |_____[long___________________1]
        |_____[long___________________2]
    
              ^                         ^
              |                         |
            build starts             build ends
    

    Which can be summarized as:

    --------------------------------------------------> timeline
    [short1]
    [short2]
    [long___________________1]
    [long___________________2]
    

    Now let's imagine that there are only 2 cores available on the machine this runs on, without a jobserver, 4 processes are started at the same time, and the OS will control parallelism by prempting them as it see fit. Assuming these are purely CPU-bound tasks, the most optimal scheduling would end up with a build time that is similar to this timeline:

    [short1][long___________________1]
    [short2][long___________________2]
    

    Now, if a jobserver is used to synchronize the parallelism, it is possible to end up in a situation where each sub-build only gets one token initially:

    [short1]
    [long___________________1]
    

    The first sub-build completes, and reuses its token to finish its second task:

    [short1][short2]
    [long___________________1]
    

    At the end of [short2] the first sub-build releases its token to the pool, but the second sub-build doesn't know about it, and waits for the completion of its first task, before launching the second one:

    [short1][short2]
    [long___________________1][long___________________2]
    

    And a long build time is the result, due to poor scheduling.
    It would be better if Ninja could detect the new token immediately to launch the second long task asap, as in:

    [short1][short2]
    [long___________________1]
                    [long___________________2]
    

    This would end up with a better build time, even though not the optimal one that can be achieved without a jobserver.

  15. digit-google commented on Oct 16, 2024

    @digit-google
    Contributor

    Some feedback, since I could roll this feature in the Ninja binary used by the Fuchsia build, and changed our builder configurations to use it.

    The Fuchsia CI bots build around 90 different build configurations of the project, only a few of them actually launch Ninja sub-builds from the top-level one (to generate SDKs that require binaries from different CPU architectures and API levels, then merging the results into a final SDK archive, in case you're curious :-))

    I could enable the jobserver feature for all build configurations and perform clean build time comparisons:

    • One SDK build configuration saves 12 minutes (from 1h to 48 minutes) due to it.
    • Other SDK build configurations save only 6 minutes (e.g. from 42 to 36 minutes), which is still a significant win.
    • Other build configurations, which do not use sub-builds, are simply not affected (as expected, but it's still nice to check).

    These confirm the savings I could observe locally on my own workstation (these are builds running on CI bots of various CPU/RAM dimensions, with remote builds enabled to boot, so we run everything with -j CPU_COUNT*10 on these).

    The jobserver feature is now enabled by default on all Fuchsia SDK build configurations.

    I addressed the few review comments on the PR #2506 that were sent, and would welcome any more feedback.

    I would be grateful if anyone could use the latest binaries generated by the Github CI and see if this improves their own multi-build times, or report issues they are facing.

    Regarding the sub-optimal scheduling described in my previous comment. Solving this will require changing the way Subprocess::DoWork() works in order to either detect new types of events (and report them). I think this is better left for another PR and discussion. What do you think?

  16. thesamesam commented on Oct 17, 2024

    @thesamesam
    Contributor

    I would be grateful if anyone could use the latest binaries generated by the Github CI and see if this improves their own multi-build times, or report issues they are facing.

    I've been using it for the last week or so. The only quirk was the interaction between NINJA_JOBSERVER=1 and --jobserver, but that wasn't a big deal at all, I just hadn't clocked it at first. I've been able to increase parallelism and not have LTO bomb the system on large world rebuilds.

    I think this is better left for another PR and discussion. What do you think?

    Yeah, I'd hate to see it delayed further, especially given nobody so far (hopefully more can test) have raised issues with the changes. It also means that we're not fighting for interest/attention on a topic that deserves its own space, I think?

  17. jhasse commented on May 17, 2025

    @jhasse
    Collaborator

    Thanks to @stefanb2, @hundeboll, @mcprat, @digit-google and everyone else who helped over the years :)

  18. added theissue type on May 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      ⌂ 换网址