From dec163fe835f69a8bcd965606ca125e1d5966bcd Mon Sep 17 00:00:00 2001 From: Andy Ross Date: Mon, 21 May 2018 10:12:59 -0700 Subject: [PATCH] sanitycheck: Disable parallel builds with Ninja Ninja and GNU make don't play well with each other. Both try to start enough processes to keep the system's CPUs busy, resulting in an O(N^2) system load in the number of processors. Long term, Ninja seems likely to support the GNU make jobserver mechanism for sharing access to parallelism. But for now we can get 90% of the way there with a simple hack: just run ninja in serial mode with -j1. Sanitycheck when run in non-trivial circumstances has PLENTY of parallelism just from the number of test cases. One interesting note is that even with -j1, system loads under ninja are rather higher. That may be because of significant work done in the (serial) makefiles that dilutes the parallelism of the eventual build, or possibly because ninja itself is multithreaded in its setup code. So I tweaked the number of jobs down to keep the load roughly where it is with make. With this change, I see no difference in behavior or system load, and a ~24% improvement in runtime. Signed-off-by: Andy Ross --- scripts/sanitycheck | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/scripts/sanitycheck b/scripts/sanitycheck index f837dc0c005..36e6e536a52 100755 --- a/scripts/sanitycheck +++ b/scripts/sanitycheck @@ -913,7 +913,7 @@ class MakeGenerator: if options.ninja: generator = "Ninja" - generator_cmd = "ninja" + generator_cmd = "ninja -j1" verb = "-v" if VERBOSE else "" else: generator = "Unix Makefiles" @@ -1154,8 +1154,16 @@ class MakeGenerator: tf.write("all: %s\n" % (" ".join(self.goals.keys()))) tf.flush() + # Normally spawn 2x as many processes as CPUs. Ninja, + # though, seems to have significant parallelism internally + # and results in rather high loads, so use 1.5x there to + # match what we see with GNU make. + loadmul = 2 + if options.ninja: + loadmul = 1.5 + cmd = ["make", "-k", "-j", - str(CPU_COUNTS * 2), "-f", tf.name, "all"] + str(int(CPU_COUNTS * loadmul)), "-f", tf.name, "all"] p = subprocess.Popen(cmd, stderr=subprocess.PIPE, stdout=devnull)