Browse Source

sanitycheck: Disable parallel builds with Ninja

Ninja and GNU make don't play well with each other.  Both try to start
enough processes to keep the system's CPUs busy, resulting in an
O(N^2) system load in the number of processors.

Long term, Ninja seems likely to support the GNU make jobserver
mechanism for sharing access to parallelism.  But for now we can get
90% of the way there with a simple hack: just run ninja in serial mode
with -j1.  Sanitycheck when run in non-trivial circumstances has
PLENTY of parallelism just from the number of test cases.

One interesting note is that even with -j1, system loads under ninja
are rather higher.  That may be because of significant work done in
the (serial) makefiles that dilutes the parallelism of the eventual
build, or possibly because ninja itself is multithreaded in its setup
code.  So I tweaked the number of jobs down to keep the load roughly
where it is with make.

With this change, I see no difference in behavior or system load, and a
~24% improvement in runtime.

Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
pull/7751/merge
Andy Ross 7 years ago committed by Anas Nashif
parent
commit
dec163fe83
  1. 12
      scripts/sanitycheck

12
scripts/sanitycheck

@ -913,7 +913,7 @@ class MakeGenerator:
if options.ninja: if options.ninja:
generator = "Ninja" generator = "Ninja"
generator_cmd = "ninja" generator_cmd = "ninja -j1"
verb = "-v" if VERBOSE else "" verb = "-v" if VERBOSE else ""
else: else:
generator = "Unix Makefiles" generator = "Unix Makefiles"
@ -1154,8 +1154,16 @@ class MakeGenerator:
tf.write("all: %s\n" % (" ".join(self.goals.keys()))) tf.write("all: %s\n" % (" ".join(self.goals.keys())))
tf.flush() tf.flush()
# Normally spawn 2x as many processes as CPUs. Ninja,
# though, seems to have significant parallelism internally
# and results in rather high loads, so use 1.5x there to
# match what we see with GNU make.
loadmul = 2
if options.ninja:
loadmul = 1.5
cmd = ["make", "-k", "-j", cmd = ["make", "-k", "-j",
str(CPU_COUNTS * 2), "-f", tf.name, "all"] str(int(CPU_COUNTS * loadmul)), "-f", tf.name, "all"]
p = subprocess.Popen(cmd, stderr=subprocess.PIPE, p = subprocess.Popen(cmd, stderr=subprocess.PIPE,
stdout=devnull) stdout=devnull)

Loading…
Cancel
Save