[L2Ork-dev] CI build times

[Sam Thursfield]

On Sun, Jun 21, 2020 at 7:18 PM Jonathan Wilkes <jon.w.wilkes at gmail.com> wrote:
> Hi all,
>
> CI build times are now by far the bottleneck in Purr Data development cycles.
>
> Anyone know if any part of our build system is parallelizable? I know
> Albert did some experiments with this at one point.

There are a couple of tools which could be interesting for PD's
builds. Neither would be an immediate fix for the build system, more
of an idea to work towards, but they may be interesting...

One is Meson[1], which is a replacement for Make/Automake/Autoconf.
The key difference with these tools is that it's deliberately not
Turing complete. I ported a project to Meson and it cut the
configure+build time by more than half. One thing Meson can't do (as
far as I know) is wrap another project's Makefile, so a porting effort
would need to be "inside out", starting with the core and all the
externals. I imagine there would need to be a way to automatically
port externals to Meson for that to be practical, because there are a
lot.

The other is BuildStream[2], which is a "meta build" tool, i.e. it
wraps an existing project's Makefile. The primary goal of BuildStream
is *reliable* builds, so it requires that everything is built in a
containerised sandbox, and needs a base image containing the necessary
build dependencies (this can be a rootfs tarball, a Docker image,
etc.). It could be interesting for PD as a replacement for the
toplevel Makefile and the 'tar-em-up.sh' script. Each PD external
could become a separate BuildStream element, and the BuildStream tool
would run the Makefiles for many of them in parallel. It also supports
caching the results, so the CI infrastructure could cache successful
builds of the externals and only rebuild them when needed.

I have some experience with both these tools if you have any questions :)
Sam

1. https://mesonbuild.com/
2. https://buildstream.build/   -- Note that there is a stable 1.4
version and a forever-in-development 2.0 version