<div dir="ltr"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jun 22, 2020 at 11:18 AM Sam Thursfield <<a href="mailto:ssssam@gmail.com">ssssam@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> Anyone know if any part of our build system is parallelizable? I know<br>
> Albert did some experiments with this at one point.<br></blockquote><div><br></div><div>Yeah, I fiddled around with this some time ago in 2018, you can still find that stuff in my parallel-builds branch on my Bitbucket (<a href="https://bitbucket.org/agraef/purr-data/branch/parallel-builds">https://bitbucket.org/agraef/purr-data/branch/parallel-builds</a>). But IIRC it never became really usable due to the complicated interdependencies between externals, so I decided that it wasn't really worth the effort.</div><div><br></div><div>One thing that works really well, though, is building Gem in parallel which reduces build times dramatically. This is in the current build system, you can trigger it by invoking `make` in the toplevel source directory like so:</div><div><br></div><div>make GEM_MAKEFLAGS=-j8</div><div><br></div><div>I do all my local package builds on Manjaro that way. If you're using `make` in the Gitlab CI runners and you have enough cpus (adjust the option to `-j` accordingly) then just adding that definition should do wonders. :)<br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
There are a couple of tools which could be interesting for PD's<br>
builds. Neither would be an immediate fix for the build system, more<br>
of an idea to work towards, but they may be interesting...<br></blockquote><div><br></div><div>Well, the current build system actually works reasonably well. Of course, the problem is that Purr includes an abundance of externals which each use their own custom build systems. Some use autoconf, some the pdlibbuilder stuff, others plain make, etc. Most of our build logic to orchestrate that big mess is currently in the tar_em_up.sh script. Getting all that under one umbrella, while also properly dealing with the interdependencies and platform dependencies, will be a monumental task to say the least. Also, for some really big externals like Gem you *want* to keep the existing build system (autoconf in that case) to facilitate compatibility with upstream.</div><div><br></div><div> Those obstacles remain no matter what you're going to use. I know autoconf, cmake, and meson reasonably well, and they would probably all be up to the job. But there's no magic wand in any of these build tools that will just make the complexities go away. If someone wants to take it on, I'm all for it! Be warned, though, that it's a *lot* of utterly tedious work, and needs to be tested *very* carefully on *all* the platforms we support. I certainly wouldn't want to be the poor sod who has to do it. ;-)</div><div><br></div><div>My 0.02 cents...</div><div><br></div><div>Albert</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
One is Meson[1], which is a replacement for Make/Automake/Autoconf.<br>
The key difference with these tools is that it's deliberately not<br>
Turing complete. I ported a project to Meson and it cut the<br>
configure+build time by more than half. One thing Meson can't do (as<br>
far as I know) is wrap another project's Makefile, so a porting effort<br>
would need to be "inside out", starting with the core and all the<br>
externals. I imagine there would need to be a way to automatically<br>
port externals to Meson for that to be practical, because there are a<br>
lot.<br>
<br>
The other is BuildStream[2], which is a "meta build" tool, i.e. it<br>
wraps an existing project's Makefile. The primary goal of BuildStream<br>
is *reliable* builds, so it requires that everything is built in a<br>
containerised sandbox, and needs a base image containing the necessary<br>
build dependencies (this can be a rootfs tarball, a Docker image,<br>
etc.). It could be interesting for PD as a replacement for the<br>
toplevel Makefile and the 'tar-em-up.sh' script. Each PD external<br>
could become a separate BuildStream element, and the BuildStream tool<br>
would run the Makefiles for many of them in parallel. It also supports<br>
caching the results, so the CI infrastructure could cache successful<br>
builds of the externals and only rebuild them when needed.<br>
<br>
I have some experience with both these tools if you have any questions :)<br>
Sam<br>
<br>
1. <a href="https://mesonbuild.com/" rel="noreferrer" target="_blank">https://mesonbuild.com/</a><br>
2. <a href="https://buildstream.build/" rel="noreferrer" target="_blank">https://buildstream.build/</a> -- Note that there is a stable 1.4<br>
version and a forever-in-development 2.0 version<br>
_______________________________________________<br>
L2Ork-dev mailing list<br>
<a href="mailto:L2Ork-dev@disis.music.vt.edu" target="_blank">L2Ork-dev@disis.music.vt.edu</a><br>
<a href="https://disis.music.vt.edu/listinfo/l2ork-dev" rel="noreferrer" target="_blank">https://disis.music.vt.edu/listinfo/l2ork-dev</a></blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Dr. Albert Gr"af<br>Computer Music Research Group, JGU Mainz, Germany<br>Email: <a href="mailto:aggraef@gmail.com" target="_blank">aggraef@gmail.com</a>, web: <a href="https://agraef.github.io/" target="_blank">https://agraef.github.io/</a></div></div></div></div></div></div></div>