[L2Ork-dev] CI build times

[Albert Graef]

On Mon, Jun 22, 2020 at 11:18 AM Sam Thursfield <ssssam at gmail.com> wrote:

> > Anyone know if any part of our build system is parallelizable? I know
> > Albert did some experiments with this at one point.
>

Yeah, I fiddled around with this some time ago in 2018, you can still find
that stuff in my parallel-builds branch on my Bitbucket (
https://bitbucket.org/agraef/purr-data/branch/parallel-builds). But IIRC it
never became really usable due to the complicated interdependencies between
externals, so I decided that it wasn't really worth the effort.

One thing that works really well, though, is building Gem in parallel which
reduces build times dramatically. This is in the current build system, you
can trigger it by invoking `make` in the toplevel source directory like so:

make GEM_MAKEFLAGS=-j8

I do all my local package builds on Manjaro that way. If you're using
`make` in the Gitlab CI runners and you have enough cpus (adjust the option
to `-j` accordingly) then just adding that definition should do wonders. :)

There are a couple of tools which could be interesting for PD's
> builds. Neither would be an immediate fix for the build system, more
> of an idea to work towards, but they may be interesting...
>

Well, the current build system actually works reasonably well. Of course,
the problem is that Purr includes an abundance of externals which each use
their own custom build systems. Some use autoconf, some the pdlibbuilder
stuff, others plain make, etc. Most of our build logic to orchestrate that
big mess is currently in the tar_em_up.sh script. Getting all that under
one umbrella, while also properly dealing with the interdependencies and
platform dependencies, will be a monumental task to say the least. Also,
for some really big externals like Gem you *want* to keep the existing
build system (autoconf in that case) to facilitate compatibility with
upstream.

Those obstacles remain no matter what you're going to use. I know autoconf,
cmake, and meson reasonably well, and they would probably all be up to the
job. But there's no magic wand in any of these build tools that will just
make the complexities go away. If someone wants to take it on, I'm all for
it! Be warned, though, that it's a *lot* of utterly tedious work, and needs
to be tested *very* carefully on *all* the platforms we support. I
certainly wouldn't want to be the poor sod who has to do it. ;-)

My 0.02 cents...

Albert


> One is Meson[1], which is a replacement for Make/Automake/Autoconf.
> The key difference with these tools is that it's deliberately not
> Turing complete. I ported a project to Meson and it cut the
> configure+build time by more than half. One thing Meson can't do (as
> far as I know) is wrap another project's Makefile, so a porting effort
> would need to be "inside out", starting with the core and all the
> externals. I imagine there would need to be a way to automatically
> port externals to Meson for that to be practical, because there are a
> lot.
>
> The other is BuildStream[2], which is a "meta build" tool, i.e. it
> wraps an existing project's Makefile. The primary goal of BuildStream
> is *reliable* builds, so it requires that everything is built in a
> containerised sandbox, and needs a base image containing the necessary
> build dependencies (this can be a rootfs tarball, a Docker image,
> etc.). It could be interesting for PD as a replacement for the
> toplevel Makefile and the 'tar-em-up.sh' script. Each PD external
> could become a separate BuildStream element, and the BuildStream tool
> would run the Makefiles for many of them in parallel. It also supports
> caching the results, so the CI infrastructure could cache successful
> builds of the externals and only rebuild them when needed.
>
> I have some experience with both these tools if you have any questions :)
> Sam
>
> 1. https://mesonbuild.com/
> 2. https://buildstream.build/   -- Note that there is a stable 1.4
> version and a forever-in-development 2.0 version
> _______________________________________________
> L2Ork-dev mailing list
> L2Ork-dev at disis.music.vt.edu
> https://disis.music.vt.edu/listinfo/l2ork-dev



-- 
Dr. Albert Gr"af
Computer Music Research Group, JGU Mainz, Germany
Email: aggraef at gmail.com, web: https://agraef.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://disis.music.vt.edu/pipermail/l2ork-dev/attachments/20200730/61e78861/attachment.html>